2024-03-04T15:57:33.716 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:57:33.716 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_suriarte@teuthology --archive /home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264 --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} --name suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi -- /home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264/orig.config.yaml 2024-03-04T15:57:33.750 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:57:33.848 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264 branch: main description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} email: null first_in_suite: false job_id: '7581264' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rgw: 20 debug rgw lifecycle: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: main sha1: a5335f5f0d4a33387826931c0d9be1a7ec43d970 owner: scheduled_suriarte@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 sleep_before_teardown: 0 suite: rgw suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: a5335f5f0d4a33387826931c0d9be1a7ec43d970 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:42:14 tube: smithi user: suriarte verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T15:57:33.848 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa; will attempt to use it 2024-03-04T15:57:33.849 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks 2024-03-04T15:57:33.849 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:57:33.851 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:57:33.877 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1fb5d10608dccee73edd6ef8a2c7b804a097c324' 2024-03-04T15:57:33.877 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:57:33.877 INFO:teuthology.packaging:ref: None 2024-03-04T15:57:33.877 INFO:teuthology.packaging:tag: None 2024-03-04T15:57:33.878 INFO:teuthology.packaging:branch: main 2024-03-04T15:57:33.878 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-04T15:57:33.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-04T15:57:34.146 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1820.g7280a241 2024-03-04T15:57:34.147 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:57:34.153 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:57:34.153 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:57:34.158 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:57:34.171 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:57:34.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:57:34.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:51:35.768268', 'locked_by': 'scheduled_suriarte@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU='} 2024-03-04T15:57:34.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:51:35.766992', 'locked_by': 'scheduled_suriarte@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI='} 2024-03-04T15:57:34.247 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:57:34.253 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-03-04T15:57:34.253 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-03-04T15:57:34.253 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:57:34.344 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9fe230f3a0>, signals=[15]) 2024-03-04T15:57:34.344 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:57:34.349 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:57:34.350 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T15:57:34.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:57:34.468 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T15:57:34.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:57:34.586 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:57:34.591 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-03-04T15:57:34.609 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-03-04T15:57:34.610 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-03-04T15:57:34.667 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:57:34.668 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:57:34.669 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-03-04T15:57:34.706 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-04T15:57:34.724 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-04T15:57:34.724 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:57:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:57:34.782 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:57:34.782 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:57:34.782 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:57:34.782 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-04T15:57:34.814 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:57:34.826 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:57:34.831 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:57:34.831 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:57:34.834 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:57:34.850 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:57:34.856 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:57:34.856 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:57:34.891 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:57:34.928 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:57:35.003 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:57:35.021 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:57:35.022 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:57:35.022 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:57:35.022 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:57:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:57:35.040 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-04T15:57:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:57:35.041 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-04T15:57:35.097 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:57:35.097 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2024-03-04T15:57:35.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-03-04T15:57:35.098 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:57:35.099 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-361.el9.x86_64 2024-03-04T15:57:35.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-03-04T15:57:37.252 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T15:57:37.308 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T15:57:38.786 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-04T15:57:38.791 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-04T15:57:39.610 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-04T15:57:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-04T15:57:42.278 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-04T15:57:46.506 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-04T15:57:47.863 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 15 kB 00:00 2024-03-04T15:57:49.540 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-03-04T15:57:54.589 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-04T15:57:56.056 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-04T15:57:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:57:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:57:56.962 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T15:57:56.966 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T15:57:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-04T15:57:56.966 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-03-04T15:57:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:57:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 100 M 2024-03-04T15:57:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 123 M 2024-03-04T15:57:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T15:57:57.383 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-04T15:57:57.633 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 39 MB/s | 21 MB 00:00 2024-03-04T15:57:58.083 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 40 MB/s | 39 MB 00:00 2024-03-04T15:57:58.391 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 33 MB/s | 33 MB 00:01 2024-03-04T15:57:58.394 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:57:58.395 INFO:teuthology.orchestra.run.smithi027.stdout:Total 70 MB/s | 100 MB 00:01 2024-03-04T15:57:59.320 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T15:57:59.770 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T15:57:59.771 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T15:58:00.200 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-04T15:58:01.180 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T15:58:01.180 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T15:58:01.782 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-04T15:58:04.053 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T15:58:04.731 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:58:04.867 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:58:05.036 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 20 MB 00:03 2024-03-04T15:58:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:58:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:58:09.899 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-04T15:58:12.115 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:58:12.115 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:58:12.218 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:58:12.225 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:58:12.226 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 100 M 2024-03-04T15:58:12.227 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 123 M 2024-03-04T15:58:12.227 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T15:58:12.551 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-04T15:58:12.835 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 37 MB/s | 21 MB 00:00 2024-03-04T15:58:13.077 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:58:13.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:13.243 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 41 MB/s | 39 MB 00:00 2024-03-04T15:58:13.560 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 33 MB/s | 33 MB 00:01 2024-03-04T15:58:13.563 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:58:13.563 INFO:teuthology.orchestra.run.smithi071.stdout:Total 74 MB/s | 100 MB 00:01 2024-03-04T15:58:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T15:58:14.826 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T15:58:14.827 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T15:58:16.175 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T15:58:16.175 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T15:58:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:19.136 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T15:58:19.791 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:58:19.929 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:58:22.703 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:58:22.900 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:58:27.635 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:58:27.853 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:32.626 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:49.093 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:49.094 INFO:teuthology.orchestra.run.smithi027.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:58:49.094 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:58:49.185 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:58:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:58:49.582 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:58:49.985 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:58:49.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:58:49.986 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T15:58:50.497 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:58:50.685 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:58:50.685 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:58:50.685 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:58:50.685 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:58:50.686 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:58:50.686 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-03-04T15:58:50.686 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:58:50.686 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:58:50.812 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:58:50.812 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:58:50.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-03-04T15:58:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T15:58:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:58:50.862 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:58:50.862 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:58:50.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:58:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-03-04T15:58:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:58:52.987 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-03-04T15:58:52.995 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-03-04T15:58:53.012 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.yssvCcKfNT 2024-03-04T15:58:53.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yssvCcKfNT 2024-03-04T15:58:53.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.yssvCcKfNT 2024-03-04T15:58:53.227 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.yssvCcKfNT is 6KB 2024-03-04T15:58:53.276 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.yssvCcKfNT 2024-03-04T15:58:53.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:58:53.360 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:58:53.360 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:58:53.360 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:58:53.361 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:58:53.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:58:53.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-03-04T15:59:01.915 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:59:01.915 INFO:teuthology.orchestra.run.smithi071.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:59:01.915 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:59:02.000 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:59:02.460 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:59:02.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:59:02.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:59:02.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:59:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:59:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:59:02.886 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T15:59:03.412 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:59:03.564 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:59:03.565 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:59:03.565 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:59:03.565 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:59:03.565 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:59:03.565 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-03-04T15:59:03.565 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:59:03.565 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:59:03.687 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:59:03.688 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:59:03.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-03-04T15:59:03.722 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-03-04T15:59:03.723 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T15:59:03.724 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:59:03.725 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:59:03.725 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:59:03.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:59:04.315 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-03-04T15:59:05.653 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:59:05.697 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-03-04T15:59:05.699 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-04T15:59:05.715 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.9M1cAt1fR0 2024-03-04T15:59:05.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9M1cAt1fR0 2024-03-04T15:59:05.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.9M1cAt1fR0 2024-03-04T15:59:05.922 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.9M1cAt1fR0 is 6KB 2024-03-04T15:59:05.976 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.9M1cAt1fR0 2024-03-04T15:59:05.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:59:06.055 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:59:06.055 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:59:06.055 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:59:06.055 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:59:06.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:59:06.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-03-04T15:59:23.531 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-03-04T15:59:23.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T15:59:23.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:59:36.232 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-03-04T15:59:36.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T15:59:36.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:59:41.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T15:59:50.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T15:59:50.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:59:53.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T15:59:54.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T16:00:03.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T16:00:03.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:00:05.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:00:05.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:00:06.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T16:00:09.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T16:00:18.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T16:00:18.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:00:24.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:00:24.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:00:50.580 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-04T16:00:50.991 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T16:00:50.992 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T16:00:50.992 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-04T16:00:51.006 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-427.el9.x86_64 2024-03-04T16:00:51.006 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T16:00:51.006 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T16:00:51.007 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T16:00:52.007 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-03-04T16:00:52.007 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T16:00:52.076 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-03-04T16:00:52.120 DEBUG:teuthology.parallel:result is None 2024-03-04T16:01:24.026 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T16:01:42.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:01:42.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:01:42.299 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-04T16:01:42.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:01:42.685 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T16:01:42.685 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-04T16:01:42.701 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-427.el9.x86_64 2024-03-04T16:01:42.702 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T16:01:42.702 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T16:01:42.702 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T16:01:43.703 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-03-04T16:01:43.703 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T16:01:43.797 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-03-04T16:01:43.841 DEBUG:teuthology.parallel:result is None 2024-03-04T16:01:43.841 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T16:01:43.848 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T16:01:43.849 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T16:01:43.851 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T16:01:43.869 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T16:01:43.876 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T16:01:43.885 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T16:01:43.886 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T16:01:43.910 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T16:01:43.937 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T16:01:43.943 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T16:01:43.943 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T16:01:43.971 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T16:01:44.011 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T16:01:44.012 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T16:01:44.027 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T16:01:44.030 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T16:01:44.031 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T16:01:44.039 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T16:01:44.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T16:01:44.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T16:01:44.105 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T16:01:44.114 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T16:01:44.114 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T16:01:44.148 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T16:01:44.164 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T16:01:44.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T16:01:44.303 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T16:01:44.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T16:01:44.433 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:01:44.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T16:01:44.504 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T16:01:44.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T16:01:44.607 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T16:01:44.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T16:01:44.737 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:01:44.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T16:01:44.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-03-04T16:01:44.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-04T16:01:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:01:44.890 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:01:45.408 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T16:01:45.416 INFO:teuthology.task.internal:Starting timer... 2024-03-04T16:01:45.416 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T16:01:45.436 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T16:01:45.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-03-04T16:01:45.495 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-03-04T16:01:45.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-04T16:01:45.548 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-04T16:01:45.550 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T16:01:45.550 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-03-04T16:01:45.578 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-03-04T16:01:45.579 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-04T16:01:45.610 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-04T16:01:45.611 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2024-03-04T16:01:45.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:01:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:01:45.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:01:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:01:45.680 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T16:01:45.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-03-04T16:01:45.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-04T16:01:45.769 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T16:01:45.779 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T16:01:45.801 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T16:01:45.802 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T16:06:04.836 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2024-03-04T16:06:04.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:06:04.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:06:04.947 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-04T16:06:05.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T16:06:05.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T16:06:05.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T16:06:05.122 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-04T16:06:05.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T16:06:05.200 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T16:06:05.211 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T16:06:05.211 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T16:06:05.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:06:05.214 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T16:06:05.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:06:05.252 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T16:06:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T16:06:05.274 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T16:06:05.291 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T16:06:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-03-04T16:06:05.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-04T16:06:05.330 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-03-04T16:06:05.335 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-04T16:06:05.351 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T16:06:05.352 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T16:06:05.368 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T16:06:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T16:06:05.420 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.437 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:06:05.438 INFO:teuthology.run_tasks:Running task install... 2024-03-04T16:06:05.447 DEBUG:teuthology.task.install:project ceph 2024-03-04T16:06:05.447 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}} 2024-03-04T16:06:05.447 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'} 2024-03-04T16:06:05.447 INFO:teuthology.task.install:Using flavor: default 2024-03-04T16:06:05.453 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T16:06:05.453 INFO:teuthology.task.install:extra packages: [] 2024-03-04T16:06:05.453 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-03-04T16:06:05.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-04T16:06:05.456 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-03-04T16:06:05.456 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-04T16:06:05.601 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/ 2024-03-04T16:06:05.601 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1670.g1fb5d106 2024-03-04T16:06:05.673 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/ 2024-03-04T16:06:05.673 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1670.g1fb5d106 2024-03-04T16:06:05.722 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T16:06:05.722 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:06:05.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T16:06:05.751 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T16:06:05.751 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1fb5d10608dccee73edd6ef8a2c7b804a097c324/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T16:06:05.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T16:06:05.851 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/1fb5d10608dccee73edd6ef8a2c7b804a097c324/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T16:06:05.851 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:06:05.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T16:06:05.883 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T16:06:05.883 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1fb5d10608dccee73edd6ef8a2c7b804a097c324/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T16:06:05.899 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T16:06:05.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T16:06:05.966 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-03-04T16:06:05.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-03-04T16:06:06.043 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T16:06:06.073 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-03-04T16:06:06.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-03-04T16:06:06.214 INFO:teuthology.orchestra.run.smithi071.stdout:74 files removed 2024-03-04T16:06:06.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-03-04T16:06:06.329 INFO:teuthology.orchestra.run.smithi027.stdout:74 files removed 2024-03-04T16:06:06.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-03-04T16:06:07.002 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 184 kB/s | 82 kB 00:00 2024-03-04T16:06:07.008 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 258 kB/s | 82 kB 00:00 2024-03-04T16:06:07.478 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-04T16:06:07.684 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-04T16:06:07.831 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-04T16:06:08.126 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-04T16:06:08.987 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-04T16:06:09.373 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 4.5 MB/s | 6.7 MB 00:01 2024-03-04T16:06:10.520 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-04T16:06:10.856 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-04T16:06:11.630 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-04T16:06:11.724 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-04T16:06:13.806 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-04T16:06:14.130 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-04T16:06:17.759 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T16:06:18.374 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-04T16:06:19.507 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-04T16:06:19.773 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 15 kB 00:00 2024-03-04T16:06:21.475 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-03-04T16:06:22.591 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 20 MB 00:02 2024-03-04T16:06:26.508 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-04T16:06:27.608 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-04T16:06:29.137 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:06:29.150 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:29.150 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 9.8 M 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.4 M 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.3 M 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 20 M 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 25 k 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T16:06:29.151 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 745 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 501 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.1 M 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse 2024-03-04T16:06:29.152 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 45 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 131 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 162 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 320 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 298 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 99 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:06:29.153 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:29.154 INFO:teuthology.orchestra.run.smithi071.stdout:Install 30 Packages 2024-03-04T16:06:29.154 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-03-04T16:06:29.154 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:29.154 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 58 M 2024-03-04T16:06:29.154 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:06:30.300 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:06:30.317 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 9.8 M 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.4 M 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.3 M 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 20 M 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 25 k 2024-03-04T16:06:30.318 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 745 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 501 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 5.1 M 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T16:06:30.319 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 45 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 131 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 162 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 320 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 298 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 99 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T16:06:30.320 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout:Install 30 Packages 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-03-04T16:06:30.321 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:30.322 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 58 M 2024-03-04T16:06:30.322 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:06:30.600 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-radosgw-19.0.0-1670.g1fb5d106.el9. 7.0 MB/s | 9.8 MB 00:01 2024-03-04T16:06:30.642 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-selinux-19.0.0-1670.g1fb5d106.el9. 609 kB/s | 25 kB 00:00 2024-03-04T16:06:30.809 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): libcephfs2-19.0.0-1670.g1fb5d106.el9.x8 4.4 MB/s | 745 kB 00:00 2024-03-04T16:06:30.876 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): ceph-base-19.0.0-1670.g1fb5d106.el9.x86 3.2 MB/s | 5.3 MB 00:01 2024-03-04T16:06:30.935 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): libradosstriper1-19.0.0-1670.g1fb5d106. 3.9 MB/s | 501 kB 00:00 2024-03-04T16:06:31.002 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-ceph-argparse-19.0.0-1670.g1fb5 679 kB/s | 45 kB 00:00 2024-03-04T16:06:31.085 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-ceph-common-19.0.0-1670.g1fb5d1 1.5 MB/s | 131 kB 00:00 2024-03-04T16:06:31.180 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-cephfs-19.0.0-1670.g1fb5d106.el 1.7 MB/s | 162 kB 00:00 2024-03-04T16:06:31.369 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): ceph-common-19.0.0-1670.g1fb5d106.el9.x 9.2 MB/s | 20 MB 00:02 2024-03-04T16:06:31.394 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-rados-19.0.0-1670.g1fb5d106.el 1.4 MB/s | 320 kB 00:00 2024-03-04T16:06:31.428 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-rbd-19.0.0-1670.g1fb5d106.el9. 5.0 MB/s | 298 kB 00:00 2024-03-04T16:06:31.462 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-rgw-19.0.0-1670.g1fb5d106.el9. 1.5 MB/s | 99 kB 00:00 2024-03-04T16:06:31.637 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): librgw2-19.0.0-1670.g1fb5d106.el9.x86_ 6.7 MB/s | 5.1 MB 00:00 2024-03-04T16:06:31.662 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 453 kB/s | 106 kB 00:00 2024-03-04T16:06:31.679 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 739 kB/s | 159 kB 00:00 2024-03-04T16:06:31.771 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 661 kB 00:00 2024-03-04T16:06:31.797 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-04T16:06:31.813 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2024-03-04T16:06:31.838 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 215 kB 00:00 2024-03-04T16:06:31.855 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-04T16:06:31.889 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): qatlib-service-23.11.0-1.el9.x86_64.rp 449 kB/s | 34 kB 00:00 2024-03-04T16:06:32.123 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-03-04T16:06:32.158 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-radosgw-19.0.0-1670.g1fb5d106.el9. 5.8 MB/s | 9.8 MB 00:01 2024-03-04T16:06:32.204 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-selinux-19.0.0-1670.g1fb5d106.el9. 607 kB/s | 25 kB 00:00 2024-03-04T16:06:32.290 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 685 kB/s | 309 kB 00:00 2024-03-04T16:06:32.315 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-04T16:06:32.317 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): libcephfs2-19.0.0-1670.g1fb5d106.el9.x8 6.2 MB/s | 745 kB 00:00 2024-03-04T16:06:32.382 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-04T16:06:32.392 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): libradosstriper1-19.0.0-1670.g1fb5d106. 6.5 MB/s | 501 kB 00:00 2024-03-04T16:06:32.490 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-04T16:06:32.610 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): ceph-base-19.0.0-1670.g1fb5d106.el9.x86 2.5 MB/s | 5.3 MB 00:02 2024-03-04T16:06:32.683 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-04T16:06:32.718 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): python3-ceph-argparse-19.0.0-1670.g1fb5 416 kB/s | 45 kB 00:00 2024-03-04T16:06:32.741 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-04T16:06:32.775 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-03-04T16:06:32.952 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): ceph-common-19.0.0-1670.g1fb5d106.el9.x 8.1 MB/s | 20 MB 00:02 2024-03-04T16:06:32.977 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-ceph-common-19.0.0-1670.g1fb5d1 507 kB/s | 131 kB 00:00 2024-03-04T16:06:33.003 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-cephfs-19.0.0-1670.g1fb5d106.el 3.2 MB/s | 162 kB 00:00 2024-03-04T16:06:33.016 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): librbd1-19.0.0-1670.g1fb5d106.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-04T16:06:33.078 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): librgw2-19.0.0-1670.g1fb5d106.el9.x86_ 7.5 MB/s | 5.1 MB 00:00 2024-03-04T16:06:33.103 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-rbd-19.0.0-1670.g1fb5d106.el9. 2.9 MB/s | 298 kB 00:00 2024-03-04T16:06:33.129 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-rados-19.0.0-1670.g1fb5d106.el 2.1 MB/s | 320 kB 00:00 2024-03-04T16:06:33.154 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): python3-rgw-19.0.0-1670.g1fb5d106.el9. 1.3 MB/s | 99 kB 00:00 2024-03-04T16:06:33.296 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 550 kB/s | 106 kB 00:00 2024-03-04T16:06:33.321 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 380 kB/s | 44 kB 00:00 2024-03-04T16:06:33.347 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 732 kB/s | 159 kB 00:00 2024-03-04T16:06:33.407 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-04T16:06:33.430 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-04T16:06:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2024-03-04T16:06:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 215 kB 00:00 2024-03-04T16:06:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 681 kB/s | 34 kB 00:00 2024-03-04T16:06:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 655 kB/s | 43 kB 00:00 2024-03-04T16:06:33.617 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): librados2-19.0.0-1670.g1fb5d106.el9.x8 3.6 MB/s | 3.4 MB 00:00 2024-03-04T16:06:33.621 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:06:33.621 INFO:teuthology.orchestra.run.smithi071.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-04T16:06:33.622 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:06:33.659 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:06:33.660 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:06:33.681 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-04T16:06:33.781 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-04T16:06:33.807 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 948 kB/s | 309 kB 00:00 2024-03-04T16:06:33.841 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-03-04T16:06:33.908 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-04T16:06:33.974 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 838 kB 00:00 2024-03-04T16:06:34.000 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-04T16:06:34.033 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-04T16:06:34.139 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:06:34.140 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:06:34.208 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): librbd1-19.0.0-1670.g1fb5d106.el9.x86_ 15 MB/s | 3.1 MB 00:00 2024-03-04T16:06:34.658 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): librados2-19.0.0-1670.g1fb5d106.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-03-04T16:06:34.664 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:06:34.664 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-04T16:06:34.664 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:06:34.701 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:06:34.701 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:06:35.208 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:06:35.208 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:06:35.616 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:06:35.698 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T16:06:35.746 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-04T16:06:35.772 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-04T16:06:35.794 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T16:06:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T16:06:35.958 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T16:06:36.016 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T16:06:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-04T16:06:36.381 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:36.546 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 8/34 2024-03-04T16:06:36.566 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/34 2024-03-04T16:06:36.629 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/34 2024-03-04T16:06:36.658 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-04T16:06:36.693 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9. 11/34 2024-03-04T16:06:36.720 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:06:36.763 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/34 2024-03-04T16:06:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 13/34 2024-03-04T16:06:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T16:06:36.855 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-04T16:06:36.866 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 13/34 2024-03-04T16:06:36.880 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-04T16:06:36.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-04T16:06:36.911 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T16:06:36.929 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T16:06:37.028 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T16:06:37.068 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T16:06:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T16:06:37.108 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T16:06:37.141 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T16:06:37.142 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T16:06:37.170 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T16:06:37.203 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T16:06:37.486 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-04T16:06:37.520 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T16:06:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 22/34 2024-03-04T16:06:37.548 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:37.615 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 22/34 2024-03-04T16:06:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:37.645 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 23/34 2024-03-04T16:06:37.684 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T16:06:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 8/34 2024-03-04T16:06:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/34 2024-03-04T16:06:37.804 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 9/34 2024-03-04T16:06:37.833 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-04T16:06:37.877 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9. 11/34 2024-03-04T16:06:37.947 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/34 2024-03-04T16:06:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 13/34 2024-03-04T16:06:38.023 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 13/34 2024-03-04T16:06:38.051 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-04T16:06:38.137 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T16:06:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T16:06:38.168 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/34 2024-03-04T16:06:38.202 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T16:06:38.239 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/34 2024-03-04T16:06:38.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T16:06:38.275 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T16:06:38.304 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T16:06:38.307 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 27/34 2024-03-04T16:06:38.336 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T16:06:38.337 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x8 28/34 2024-03-04T16:06:38.666 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T16:06:38.805 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 22/34 2024-03-04T16:06:38.805 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 22/34 2024-03-04T16:06:38.813 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 23/34 2024-03-04T16:06:38.857 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T16:06:39.336 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T16:06:39.369 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/34 2024-03-04T16:06:39.440 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 26/34 2024-03-04T16:06:39.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 27/34 2024-03-04T16:06:39.556 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x8 28/34 2024-03-04T16:06:40.439 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:40.475 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:41.038 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:41.104 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 30/34 2024-03-04T16:06:41.195 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 30/34 2024-03-04T16:06:41.196 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T16:06:41.196 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T16:06:41.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:41.230 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:41.837 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:41.884 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:42.473 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 29/34 2024-03-04T16:06:42.512 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 30/34 2024-03-04T16:06:42.604 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 30/34 2024-03-04T16:06:42.605 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T16:06:42.605 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T16:06:42.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:42.630 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:51.053 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-03-04T16:06:51.054 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:51.964 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/34 2024-03-04T16:06:51.989 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/34 2024-03-04T16:06:51.989 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:06:51.990 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T16:06:51.990 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T16:06:51.990 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T16:06:51.990 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:51.991 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T16:06:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T16:06:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:52.527 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-03-04T16:06:52.527 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-03-04T16:06:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 6/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9. 8/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x8 9/34 2024-03-04T16:06:53.218 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 13/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T16:06:53.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T16:06:53.220 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/34 2024-03-04T16:06:53.467 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/34 2024-03-04T16:06:53.499 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/34 2024-03-04T16:06:53.499 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:06:53.499 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T16:06:53.499 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T16:06:53.500 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T16:06:53.500 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:53.501 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T16:06:53.587 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T16:06:53.587 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T16:06:54.144 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.145 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:54.146 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:06:54.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/34 2024-03-04T16:06:54.703 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_6 6/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 7/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9. 8/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x8 9/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 10/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 11/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 12/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 13/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T16:06:54.704 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T16:06:54.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 31/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T16:06:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/34 2024-03-04T16:06:55.060 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:29 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:06:55.227 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 47 M 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 163 k 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T16:06:55.228 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-03-04T16:06:55.229 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 205 M 2024-03-04T16:06:55.230 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:06:55.438 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-04T16:06:55.480 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T16:06:55.531 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.7 MB/s | 186 kB 00:00 2024-03-04T16:06:55.556 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): libcephsqlite-19.0.0-1670.g1fb5d106.el9. 502 kB/s | 163 kB 00:00 2024-03-04T16:06:55.581 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-04T16:06:56.037 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T16:06:56.037 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:56.037 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T16:06:56.038 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:06:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:56.040 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:06:56.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-03-04T16:06:56.983 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:29 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:06:57.157 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 47 M 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 163 k 2024-03-04T16:06:57.158 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-03-04T16:06:57.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:06:57.160 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 48 M 2024-03-04T16:06:57.160 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 205 M 2024-03-04T16:06:57.160 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:06:57.367 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-04T16:06:57.399 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-04T16:06:57.449 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-04T16:06:57.466 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): libcephsqlite-19.0.0-1670.g1fb5d106.el9. 534 kB/s | 163 kB 00:00 2024-03-04T16:06:57.500 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-04T16:06:57.614 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-19.0.0-1670.g1fb5d106.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-04T16:06:57.617 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:06:57.618 INFO:teuthology.orchestra.run.smithi071.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-04T16:06:57.618 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:06:57.638 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:06:57.639 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:06:58.018 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:06:58.018 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:06:58.599 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:06:58.669 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T16:06:58.703 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T16:06:58.749 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T16:06:58.792 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T16:06:58.817 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/6 2024-03-04T16:07:00.108 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): ceph-test-19.0.0-1670.g1fb5d106.el9.x86_ 16 MB/s | 47 MB 00:02 2024-03-04T16:07:00.111 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:00.112 INFO:teuthology.orchestra.run.smithi027.stdout:Total 16 MB/s | 48 MB 00:02 2024-03-04T16:07:00.112 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:00.135 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:00.135 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:00.534 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:00.535 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:01.134 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T16:07:01.250 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T16:07:01.295 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T16:07:01.342 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T16:07:01.370 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/6 2024-03-04T16:07:03.296 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/6 2024-03-04T16:07:03.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/6 2024-03-04T16:07:05.818 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/6 2024-03-04T16:07:05.818 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/6 2024-03-04T16:07:05.818 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/6 2024-03-04T16:07:05.818 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T16:07:05.819 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T16:07:05.819 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T16:07:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/6 2024-03-04T16:07:06.175 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/6 2024-03-04T16:07:06.209 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:06.210 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:06.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-03-04T16:07:07.105 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:41 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:07.279 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:07.281 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 6.6 k 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 2.1 M 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 1.5 M 2024-03-04T16:07:07.282 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 248 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 4.7 M 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 17 M 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T16:07:07.283 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T16:07:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T16:07:07.285 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:Install 36 Packages 2024-03-04T16:07:07.286 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:07.287 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2024-03-04T16:07:07.287 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 107 M 2024-03-04T16:07:07.287 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:07.849 INFO:teuthology.orchestra.run.smithi071.stdout:(1/36): ceph-19.0.0-1670.g1fb5d106.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-03-04T16:07:07.999 INFO:teuthology.orchestra.run.smithi071.stdout:(2/36): ceph-mgr-19.0.0-1670.g1fb5d106.el9.x86_ 3.1 MB/s | 1.5 MB 00:00 2024-03-04T16:07:08.082 INFO:teuthology.orchestra.run.smithi071.stdout:(3/36): ceph-mds-19.0.0-1670.g1fb5d106.el9.x86_ 3.8 MB/s | 2.1 MB 00:00 2024-03-04T16:07:08.149 INFO:teuthology.orchestra.run.smithi071.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1670.g1fb5 3.6 MB/s | 248 kB 00:00 2024-03-04T16:07:08.317 INFO:teuthology.orchestra.run.smithi071.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-04T16:07:08.375 INFO:teuthology.orchestra.run.smithi071.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-04T16:07:08.417 INFO:teuthology.orchestra.run.smithi071.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-03-04T16:07:08.459 INFO:teuthology.orchestra.run.smithi071.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-04T16:07:08.501 INFO:teuthology.orchestra.run.smithi071.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 927 kB/s | 39 kB 00:00 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 6/6 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/6 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/6 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T16:07:08.539 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T16:07:08.559 INFO:teuthology.orchestra.run.smithi071.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-04T16:07:08.643 INFO:teuthology.orchestra.run.smithi071.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-04T16:07:08.685 INFO:teuthology.orchestra.run.smithi071.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-04T16:07:08.727 INFO:teuthology.orchestra.run.smithi071.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-04T16:07:08.760 INFO:teuthology.orchestra.run.smithi071.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-03-04T16:07:08.811 INFO:teuthology.orchestra.run.smithi071.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-04T16:07:08.894 INFO:teuthology.orchestra.run.smithi071.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-16.el9.x86_64 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:08.939 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T16:07:08.940 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T16:07:08.940 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T16:07:08.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:08.940 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:09.145 INFO:teuthology.orchestra.run.smithi071.stdout:(17/36): ceph-mon-19.0.0-1670.g1fb5d106.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-03-04T16:07:09.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-03-04T16:07:09.320 INFO:teuthology.orchestra.run.smithi071.stdout:(18/36): ceph-osd-19.0.0-1670.g1fb5d106.el9.x86 13 MB/s | 17 MB 00:01 2024-03-04T16:07:09.345 INFO:teuthology.orchestra.run.smithi071.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 335 kB/s | 151 kB 00:00 2024-03-04T16:07:09.545 INFO:teuthology.orchestra.run.smithi071.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-04T16:07:09.570 INFO:teuthology.orchestra.run.smithi071.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 102 kB/s | 43 kB 00:00 2024-03-04T16:07:09.638 INFO:teuthology.orchestra.run.smithi071.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-04T16:07:09.729 INFO:teuthology.orchestra.run.smithi071.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-04T16:07:09.746 INFO:teuthology.orchestra.run.smithi071.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-04T16:07:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-04T16:07:09.791 INFO:teuthology.orchestra.run.smithi071.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 365 kB/s | 172 kB 00:00 2024-03-04T16:07:09.842 INFO:teuthology.orchestra.run.smithi071.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 208 kB/s | 20 kB 00:00 2024-03-04T16:07:09.859 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:42 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:09.874 INFO:teuthology.orchestra.run.smithi071.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 417 kB/s | 46 kB 00:00 2024-03-04T16:07:09.900 INFO:teuthology.orchestra.run.smithi071.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 724 kB/s | 79 kB 00:00 2024-03-04T16:07:09.967 INFO:teuthology.orchestra.run.smithi071.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-04T16:07:09.992 INFO:teuthology.orchestra.run.smithi071.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-04T16:07:10.017 INFO:teuthology.orchestra.run.smithi071.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 768 kB/s | 90 kB 00:00 2024-03-04T16:07:10.035 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:10.036 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================= 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================= 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 6.6 k 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 2.1 M 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 1.5 M 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 248 k 2024-03-04T16:07:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 4.7 M 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 17 M 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T16:07:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T16:07:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:10.040 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================= 2024-03-04T16:07:10.041 INFO:teuthology.orchestra.run.smithi027.stdout:Install 36 Packages 2024-03-04T16:07:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:10.041 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 30 M 2024-03-04T16:07:10.041 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 107 M 2024-03-04T16:07:10.041 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:10.059 INFO:teuthology.orchestra.run.smithi071.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-04T16:07:10.159 INFO:teuthology.orchestra.run.smithi071.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-04T16:07:10.185 INFO:teuthology.orchestra.run.smithi071.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-04T16:07:10.211 INFO:teuthology.orchestra.run.smithi071.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-03-04T16:07:10.214 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:10.214 INFO:teuthology.orchestra.run.smithi071.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-04T16:07:10.232 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:10.274 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:10.275 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:10.430 INFO:teuthology.orchestra.run.smithi027.stdout:(1/36): ceph-19.0.0-1670.g1fb5d106.el9.x86_64.r 33 kB/s | 6.6 kB 00:00 2024-03-04T16:07:10.605 INFO:teuthology.orchestra.run.smithi027.stdout:(2/36): ceph-mds-19.0.0-1670.g1fb5d106.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-03-04T16:07:10.673 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:10.674 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:(3/36): ceph-mgr-19.0.0-1670.g1fb5d106.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-04T16:07:11.034 INFO:teuthology.orchestra.run.smithi027.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1670.g1fb5 1.9 MB/s | 248 kB 00:00 2024-03-04T16:07:11.189 INFO:teuthology.orchestra.run.smithi027.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-04T16:07:11.248 INFO:teuthology.orchestra.run.smithi027.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-04T16:07:11.290 INFO:teuthology.orchestra.run.smithi027.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-04T16:07:11.332 INFO:teuthology.orchestra.run.smithi027.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-04T16:07:11.374 INFO:teuthology.orchestra.run.smithi027.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-04T16:07:11.458 INFO:teuthology.orchestra.run.smithi027.stdout:(10/36): ceph-mon-19.0.0-1670.g1fb5d106.el9.x86 4.6 MB/s | 4.7 MB 00:01 2024-03-04T16:07:11.483 INFO:teuthology.orchestra.run.smithi027.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-04T16:07:11.650 INFO:teuthology.orchestra.run.smithi027.stdout:(12/36): ceph-osd-19.0.0-1670.g1fb5d106.el9.x86 16 MB/s | 17 MB 00:01 2024-03-04T16:07:11.683 INFO:teuthology.orchestra.run.smithi027.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-03-04T16:07:11.709 INFO:teuthology.orchestra.run.smithi027.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 457 kB/s | 103 kB 00:00 2024-03-04T16:07:11.734 INFO:teuthology.orchestra.run.smithi027.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-04T16:07:11.759 INFO:teuthology.orchestra.run.smithi027.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-03-04T16:07:11.784 INFO:teuthology.orchestra.run.smithi027.stdout:(17/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-04T16:07:11.868 INFO:teuthology.orchestra.run.smithi027.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-04T16:07:11.902 INFO:teuthology.orchestra.run.smithi027.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-03-04T16:07:11.952 INFO:teuthology.orchestra.run.smithi027.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 786 kB/s | 151 kB 00:00 2024-03-04T16:07:11.994 INFO:teuthology.orchestra.run.smithi027.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-04T16:07:12.011 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:12.019 INFO:teuthology.orchestra.run.smithi027.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-04T16:07:12.044 INFO:teuthology.orchestra.run.smithi027.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-04T16:07:12.070 INFO:teuthology.orchestra.run.smithi027.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-04T16:07:12.079 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-04T16:07:12.095 INFO:teuthology.orchestra.run.smithi027.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-04T16:07:12.112 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-04T16:07:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-03-04T16:07:12.139 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-04T16:07:12.145 INFO:teuthology.orchestra.run.smithi027.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-04T16:07:12.170 INFO:teuthology.orchestra.run.smithi027.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-04T16:07:12.196 INFO:teuthology.orchestra.run.smithi027.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-04T16:07:12.210 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-04T16:07:12.224 INFO:teuthology.orchestra.run.smithi027.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-04T16:07:12.246 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-04T16:07:12.247 INFO:teuthology.orchestra.run.smithi027.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 783 kB/s | 79 kB 00:00 2024-03-04T16:07:12.272 INFO:teuthology.orchestra.run.smithi027.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-03-04T16:07:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-04T16:07:12.297 INFO:teuthology.orchestra.run.smithi027.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 894 kB/s | 90 kB 00:00 2024-03-04T16:07:12.330 INFO:teuthology.orchestra.run.smithi027.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-04T16:07:12.355 INFO:teuthology.orchestra.run.smithi027.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-04T16:07:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-04T16:07:12.381 INFO:teuthology.orchestra.run.smithi027.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-04T16:07:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 30 MB 00:02 2024-03-04T16:07:12.403 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:12.452 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:12.452 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:12.465 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-04T16:07:12.507 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-04T16:07:12.578 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-04T16:07:12.632 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-04T16:07:12.670 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-04T16:07:12.728 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-04T16:07:12.784 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-04T16:07:12.894 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:12.894 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:12.968 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-04T16:07:13.016 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-04T16:07:13.047 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-04T16:07:13.081 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-04T16:07:13.112 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-04T16:07:13.213 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-04T16:07:13.247 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T16:07:13.280 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-04T16:07:13.320 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-04T16:07:13.356 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-04T16:07:13.390 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-04T16:07:13.479 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-04T16:07:13.513 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-04T16:07:13.584 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-04T16:07:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-04T16:07:13.847 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T16:07:14.021 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9. 31/36 2024-03-04T16:07:14.051 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/36 2024-03-04T16:07:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/36 2024-03-04T16:07:14.076 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:14.076 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T16:07:14.076 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T16:07:14.076 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T16:07:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:14.248 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:14.309 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-04T16:07:14.352 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-04T16:07:14.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-04T16:07:14.462 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-04T16:07:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-04T16:07:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-04T16:07:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-04T16:07:14.725 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-04T16:07:14.771 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-04T16:07:14.854 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-04T16:07:14.906 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-04T16:07:14.944 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-04T16:07:15.036 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-04T16:07:15.102 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-04T16:07:15.301 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-04T16:07:15.347 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-04T16:07:15.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-04T16:07:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-04T16:07:15.450 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-04T16:07:15.501 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/36 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/36 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T16:07:15.527 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:15.528 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-04T16:07:15.569 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T16:07:15.602 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-04T16:07:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-04T16:07:15.686 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-04T16:07:15.720 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-04T16:07:15.814 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-04T16:07:15.852 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-04T16:07:15.926 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-04T16:07:15.969 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 34/36 2024-03-04T16:07:15.991 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 34/36 2024-03-04T16:07:15.992 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:15.992 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T16:07:15.992 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T16:07:15.992 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T16:07:15.992 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-04T16:07:16.203 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 35/36 2024-03-04T16:07:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T16:07:16.225 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 35/36 2024-03-04T16:07:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T16:07:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T16:07:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T16:07:16.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:16.292 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 36/36 2024-03-04T16:07:16.375 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9. 31/36 2024-03-04T16:07:16.405 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/36 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 32/36 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T16:07:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/36 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 33/36 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T16:07:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:18.025 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 36/36 2024-03-04T16:07:18.025 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9. 6/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-04T16:07:18.026 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-04T16:07:18.027 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-04T16:07:18.028 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-04T16:07:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 34/36 2024-03-04T16:07:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 34/36 2024-03-04T16:07:18.465 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:18.465 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T16:07:18.466 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T16:07:18.466 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T16:07:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 35/36 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 35/36 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T16:07:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 36/36 2024-03-04T16:07:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-04T16:07:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T16:07:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T16:07:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:18.990 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:19.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-03-04T16:07:19.917 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:53 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:20.031 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:07:20.084 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:20.085 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:07:20.085 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:20.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 36/36 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/36 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/36 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 3/36 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 4/36 2024-03-04T16:07:20.634 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 5/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9. 6/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-04T16:07:20.635 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-04T16:07:20.636 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-04T16:07:20.637 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-04T16:07:20.638 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-04T16:07:20.704 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:54 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:20.869 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 748 k 2024-03-04T16:07:20.870 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 748 k 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 754 k 2024-03-04T16:07:20.871 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:21.176 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.0.0-1670.g1fb5d106.el9.noarch.rpm 2.4 MB/s | 748 kB 00:00 2024-03-04T16:07:21.177 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:21.177 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.4 MB/s | 748 kB 00:00 2024-03-04T16:07:21.177 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:21.182 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:21.182 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:21.188 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:21.188 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:21.292 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:21.699 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:21.782 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T16:07:21.783 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T16:07:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T16:07:21.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:21.785 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:21.786 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:22.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-03-04T16:07:22.154 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:22.468 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:22.469 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:22.469 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:22.469 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:22.469 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:22.469 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:22.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T16:07:22.711 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:55 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:22.828 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:07:22.883 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:22.884 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:07:22.884 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:22.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-03-04T16:07:23.244 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:23.414 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:23.414 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 145 k 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 436 k 2024-03-04T16:07:23.415 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:23.532 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:56 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:23.630 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.0.0-1670.g1fb5d1 678 kB/s | 145 kB 00:00 2024-03-04T16:07:23.631 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:23.631 INFO:teuthology.orchestra.run.smithi071.stdout:Total 673 kB/s | 145 kB 00:00 2024-03-04T16:07:23.631 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:23.638 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:23.638 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:23.690 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:23.691 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:23.706 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 748 k 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 748 k 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 754 k 2024-03-04T16:07:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:23.922 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:23.922 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T16:07:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:24.006 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-19.0.0-1670.g1fb5d106.el9.noarch.rpm 2.4 MB/s | 748 kB 00:00 2024-03-04T16:07:24.007 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:24.007 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.4 MB/s | 748 kB 00:00 2024-03-04T16:07:24.007 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:24.013 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:24.013 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:24.019 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:24.019 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:24.960 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:24.960 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:24.961 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:24.961 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:24.961 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:24.961 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:25.040 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:25.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-03-04T16:07:25.360 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/1 2024-03-04T16:07:25.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:25.360 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:25.361 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:25.361 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:25.361 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:25.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T16:07:25.802 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:59 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:25.915 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:07:25.967 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:25.968 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:07:25.969 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:26.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T16:07:26.137 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:59 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:26.312 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 145 k 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 145 k 2024-03-04T16:07:26.314 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 436 k 2024-03-04T16:07:26.314 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:26.539 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-19.0.0-1670.g1fb5d1 646 kB/s | 145 kB 00:00 2024-03-04T16:07:26.539 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:26.539 INFO:teuthology.orchestra.run.smithi027.stdout:Total 641 kB/s | 145 kB 00:00 2024-03-04T16:07:26.540 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:26.548 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:26.548 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:26.585 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:00 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:26.603 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:26.603 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:26.752 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:26.753 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 1.8 M 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 31 k 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 15 k 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T16:07:26.754 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-04T16:07:26.755 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:26.756 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.9 M 2024-03-04T16:07:26.756 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-04T16:07:26.756 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:26.814 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:26.840 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:26.840 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T16:07:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:27.150 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1670.g1f 172 kB/s | 31 kB 00:00 2024-03-04T16:07:27.167 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1670.g1fb 77 kB/s | 15 kB 00:00 2024-03-04T16:07:27.409 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 984 kB/s | 238 kB 00:00 2024-03-04T16:07:27.484 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-03-04T16:07:27.593 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-03-04T16:07:27.827 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106. 1/1 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:27.855 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:27.943 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1670.g1fb5d10 1.8 MB/s | 1.8 MB 00:00 2024-03-04T16:07:28.052 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 968 kB/s | 548 kB 00:00 2024-03-04T16:07:28.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-03-04T16:07:28.152 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-04T16:07:28.252 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-04T16:07:28.361 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 346 kB/s | 144 kB 00:00 2024-03-04T16:07:28.411 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-03-04T16:07:28.414 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:28.414 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-04T16:07:28.432 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:28.449 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:28.449 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:28.626 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:28.626 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:28.667 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:01 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:28.787 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:07:28.842 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:28.843 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:07:28.843 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:28.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T16:07:29.149 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:29.228 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-04T16:07:29.271 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-04T16:07:29.317 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-04T16:07:29.349 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-04T16:07:29.432 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-04T16:07:29.497 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:02 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:29.557 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-04T16:07:29.651 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-04T16:07:29.671 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================== 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================== 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 1.8 M 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 31 k 2024-03-04T16:07:29.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 15 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================== 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-03-04T16:07:29.673 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:29.674 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 6.9 M 2024-03-04T16:07:29.674 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-03-04T16:07:29.674 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:29.694 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-04T16:07:29.723 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-04T16:07:29.755 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9 10/12 2024-03-04T16:07:29.994 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el 11/12 2024-03-04T16:07:30.038 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 12/12 2024-03-04T16:07:30.306 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1670.g1fb 61 kB/s | 15 kB 00:00 2024-03-04T16:07:30.310 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1670.g1f 118 kB/s | 31 kB 00:00 2024-03-04T16:07:30.507 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-04T16:07:30.536 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-03-04T16:07:30.578 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1670.g1fb5d10 3.3 MB/s | 1.8 MB 00:00 2024-03-04T16:07:30.737 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-04T16:07:30.770 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 136 kB/s | 19 kB 00:00 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 12/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el 1/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 2/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9 3/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-04T16:07:30.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-04T16:07:30.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-04T16:07:30.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-04T16:07:30.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-04T16:07:30.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-04T16:07:30.887 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-04T16:07:30.912 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-04T16:07:31.071 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.1 MB/s | 2.0 MB 00:00 2024-03-04T16:07:31.096 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 147 kB/s | 31 kB 00:00 2024-03-04T16:07:31.121 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 904 kB/s | 188 kB 00:00 2024-03-04T16:07:31.128 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:31.128 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.7 MB/s | 6.9 MB 00:01 2024-03-04T16:07:31.146 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:31.164 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:31.164 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:31.201 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T16:07:31.202 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T16:07:31.203 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T16:07:31.203 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:31.203 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:31.346 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:31.347 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:31.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T16:07:31.869 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:31.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-04T16:07:31.991 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-04T16:07:32.006 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:06 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-04T16:07:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-04T16:07:32.174 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:32.174 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 7.4 M 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T16:07:32.175 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:32.176 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 209 M 2024-03-04T16:07:32.177 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:32.306 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-04T16:07:32.375 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 204 kB/s | 32 kB 00:00 2024-03-04T16:07:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-04T16:07:32.417 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 423 kB/s | 17 kB 00:00 2024-03-04T16:07:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-04T16:07:32.467 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 695 kB/s | 34 kB 00:00 2024-03-04T16:07:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-04T16:07:32.526 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.8 MB/s | 3.0 MB 00:00 2024-03-04T16:07:32.541 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9 10/12 2024-03-04T16:07:32.794 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el 11/12 2024-03-04T16:07:32.818 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-04T16:07:32.823 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 12/12 2024-03-04T16:07:32.894 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-04T16:07:32.919 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-04T16:07:33.194 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 802 kB 00:00 2024-03-04T16:07:33.420 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-04T16:07:33.444 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 783 kB/s | 194 kB 00:00 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 12/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el 1/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noa 2/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9 3/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-04T16:07:33.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-04T16:07:33.661 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T16:07:33.988 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:33.989 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:34.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T16:07:34.778 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.9 MB/s | 7.4 MB 00:02 2024-03-04T16:07:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:34.781 INFO:teuthology.orchestra.run.smithi071.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-04T16:07:34.796 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:34.823 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:34.823 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:34.844 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:07 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:35.022 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 7.4 M 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T16:07:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-03-04T16:07:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:35.025 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 37 M 2024-03-04T16:07:35.025 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 209 M 2024-03-04T16:07:35.025 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:35.185 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:35.185 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:35.386 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 164 kB/s | 32 kB 00:00 2024-03-04T16:07:35.436 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 344 kB/s | 17 kB 00:00 2024-03-04T16:07:35.495 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 593 kB/s | 34 kB 00:00 2024-03-04T16:07:35.545 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-03-04T16:07:35.812 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-04T16:07:35.896 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-04T16:07:35.896 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:35.921 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-04T16:07:35.959 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T16:07:35.990 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T16:07:36.080 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 802 kB 00:00 2024-03-04T16:07:36.086 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T16:07:36.119 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T16:07:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T16:07:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T16:07:36.355 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-04T16:07:36.380 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 652 kB/s | 194 kB 00:00 2024-03-04T16:07:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T16:07:36.905 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.3 MB/s | 7.4 MB 00:01 2024-03-04T16:07:37.061 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T16:07:37.172 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T16:07:37.372 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 241 kB/s | 245 kB 00:01 2024-03-04T16:07:37.375 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:37.375 INFO:teuthology.orchestra.run.smithi027.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-04T16:07:37.389 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:37.417 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:37.417 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:37.801 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:37.801 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:38.406 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T16:07:38.656 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:38.737 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T16:07:38.768 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T16:07:38.896 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T16:07:38.931 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T16:07:39.115 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T16:07:39.146 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T16:07:39.225 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T16:07:39.427 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T16:07:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 12/12 2024-03-04T16:07:39.934 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T16:07:40.056 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T16:07:41.395 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 12/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 1/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T16:07:42.108 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T16:07:42.109 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T16:07:42.109 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T16:07:42.472 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T16:07:42.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 12/12 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T16:07:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:42.536 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:42.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-03-04T16:07:43.470 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:17 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:43.637 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:43.638 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:43.638 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:43.638 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:43.638 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:43.638 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 49 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T16:07:43.639 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2024-03-04T16:07:43.640 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-04T16:07:43.641 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:44.158 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-04T16:07:44.183 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 132 kB/s | 24 kB 00:00 2024-03-04T16:07:44.268 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-04T16:07:44.292 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-04T16:07:44.317 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-04T16:07:44.343 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-04T16:07:44.368 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): ceph-mgr-rook-19.0.0-1670.g1fb5d106.el9 134 kB/s | 49 kB 00:00 2024-03-04T16:07:44.527 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-03-04T16:07:44.552 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-03-04T16:07:44.652 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-03-04T16:07:44.678 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 775 kB/s | 203 kB 00:00 2024-03-04T16:07:44.761 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 827 kB/s | 90 kB 00:00 2024-03-04T16:07:45.003 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-04T16:07:45.007 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:45.007 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-04T16:07:45.007 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:45.020 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:45.020 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:45.120 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 12/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d 1/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T16:07:45.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T16:07:45.122 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T16:07:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:45.627 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:45.639 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:45.808 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T16:07:45.880 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T16:07:45.935 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T16:07:45.972 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T16:07:45.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2024-03-04T16:07:46.013 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T16:07:46.107 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T16:07:46.233 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T16:07:46.289 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T16:07:46.573 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:19 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:46.748 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:46.749 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:46.749 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 49 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T16:07:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-03-04T16:07:46.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.0 M 2024-03-04T16:07:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-03-04T16:07:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:46.946 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-04T16:07:46.972 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 145 kB/s | 24 kB 00:00 2024-03-04T16:07:47.011 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T16:07:47.039 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-04T16:07:47.055 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-04T16:07:47.061 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T16:07:47.089 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-04T16:07:47.094 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T16:07:47.114 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 727 kB/s | 43 kB 00:00 2024-03-04T16:07:47.139 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): ceph-mgr-rook-19.0.0-1670.g1fb5d106.el9 147 kB/s | 49 kB 00:00 2024-03-04T16:07:47.143 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T16:07:47.164 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 13/13 2024-03-04T16:07:47.381 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-04T16:07:47.407 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-04T16:07:47.557 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-04T16:07:47.582 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 458 kB/s | 203 kB 00:00 2024-03-04T16:07:47.691 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-03-04T16:07:47.725 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-04T16:07:47.728 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:47.729 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-04T16:07:47.729 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:47.743 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:47.744 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:47.853 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:47.853 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 13/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 1/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T16:07:47.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T16:07:47.946 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T16:07:47.946 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T16:07:47.946 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T16:07:48.324 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T16:07:48.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T16:07:48.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T16:07:48.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T16:07:48.326 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:48.326 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:48.389 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T16:07:48.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T16:07:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T16:07:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T16:07:48.736 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T16:07:48.778 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T16:07:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T16:07:48.986 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T16:07:49.046 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T16:07:49.155 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:23 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:49.325 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 138 k 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:49.326 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-04T16:07:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-04T16:07:49.327 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.8 M 2024-03-04T16:07:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-03-04T16:07:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:49.813 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T16:07:49.859 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T16:07:49.893 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T16:07:49.941 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T16:07:49.979 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 13/13 2024-03-04T16:07:50.028 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1670.g1fb5d106.e 523 kB/s | 138 kB 00:00 2024-03-04T16:07:50.054 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 786 kB/s | 227 kB 00:00 2024-03-04T16:07:50.329 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-04T16:07:50.354 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-03-04T16:07:50.646 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-03-04T16:07:50.771 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 738 kB/s | 548 kB 00:00 2024-03-04T16:07:50.773 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:50.773 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-04T16:07:50.773 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:50.783 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:50.783 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:50.786 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 13/13 2024-03-04T16:07:50.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 1/13 2024-03-04T16:07:50.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T16:07:50.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T16:07:50.874 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:50.875 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:51.186 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T16:07:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T16:07:51.414 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T16:07:51.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T16:07:51.921 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T16:07:52.007 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T16:07:52.024 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:25 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:52.070 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T16:07:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 6/6 2024-03-04T16:07:52.200 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 138 k 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T16:07:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-03-04T16:07:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:52.203 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 6.8 M 2024-03-04T16:07:52.203 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 33 M 2024-03-04T16:07:52.203 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:52.693 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 556 kB/s | 227 kB 00:00 2024-03-04T16:07:52.902 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1670.g1fb5d106.e 224 kB/s | 138 kB 00:00 2024-03-04T16:07:52.931 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 6/6 2024-03-04T16:07:52.931 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/6 2024-03-04T16:07:52.931 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T16:07:52.931 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T16:07:52.931 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T16:07:52.932 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T16:07:53.069 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 5.8 MB 00:00 2024-03-04T16:07:53.202 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-04T16:07:53.244 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:53.257 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:53.378 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 166 kB/s | 51 kB 00:00 2024-03-04T16:07:53.381 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:53.382 INFO:teuthology.orchestra.run.smithi027.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-04T16:07:53.382 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:53.391 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:53.392 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:53.487 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:53.489 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:53.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-03-04T16:07:53.791 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:53.971 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T16:07:54.023 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T16:07:54.069 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:28 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:54.237 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 887 k 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:54.238 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-04T16:07:54.239 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:54.239 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 966 k 2024-03-04T16:07:54.239 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.8 M 2024-03-04T16:07:54.239 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:54.414 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 450 kB/s | 79 kB 00:00 2024-03-04T16:07:54.547 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.0.0-1670.g1fb5d106.el9.x86_ 2.8 MB/s | 887 kB 00:00 2024-03-04T16:07:54.548 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:54.549 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.0 MB/s | 966 kB 00:00 2024-03-04T16:07:54.549 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:54.569 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:54.569 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:54.582 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T16:07:54.645 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:54.645 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:54.676 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T16:07:54.745 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T16:07:54.775 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 6/6 2024-03-04T16:07:54.834 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:54.970 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T16:07:55.039 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/2 2024-03-04T16:07:55.618 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 6/6 2024-03-04T16:07:55.619 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 1/6 2024-03-04T16:07:55.619 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T16:07:55.619 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T16:07:55.619 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T16:07:55.619 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T16:07:55.709 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/2 2024-03-04T16:07:55.709 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/2 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T16:07:55.961 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T16:07:55.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T16:07:55.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T16:07:55.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T16:07:55.962 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:55.962 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:56.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T16:07:56.031 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:56.031 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:56.031 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T16:07:56.031 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:56.031 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:56.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-03-04T16:07:56.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-03-04T16:07:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:56.883 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:07:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:56.967 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 887 k 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-03-04T16:07:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:56.969 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 966 k 2024-03-04T16:07:56.969 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.8 M 2024-03-04T16:07:56.969 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:57.054 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:07:57.055 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:57.055 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-04T16:07:57.055 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:57.055 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 264 k 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 332 k 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.5 M 2024-03-04T16:07:57.056 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:07:57.129 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 491 kB/s | 79 kB 00:00 2024-03-04T16:07:57.201 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 476 kB/s | 69 kB 00:00 2024-03-04T16:07:57.296 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-19.0.0-1670.g1fb5d106.el9.x86_ 2.7 MB/s | 887 kB 00:00 2024-03-04T16:07:57.297 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:57.298 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.9 MB/s | 966 kB 00:00 2024-03-04T16:07:57.298 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:07:57.320 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:07:57.320 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:07:57.402 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:07:57.403 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:07:57.409 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-volume-19.0.0-1670.g1fb5d106.el9.no 748 kB/s | 264 kB 00:00 2024-03-04T16:07:57.410 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:07:57.411 INFO:teuthology.orchestra.run.smithi071.stdout:Total 936 kB/s | 332 kB 00:00 2024-03-04T16:07:57.411 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:07:57.419 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:07:57.419 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:07:57.459 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:07:57.460 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:07:57.596 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:07:57.611 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:07:57.748 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T16:07:57.790 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T16:07:57.825 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/2 2024-03-04T16:07:57.829 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2/2 2024-03-04T16:07:57.844 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2/2 2024-03-04T16:07:57.844 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:07:57.844 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T16:07:57.844 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:58.496 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2/2 2024-03-04T16:07:58.496 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/2 2024-03-04T16:07:58.621 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 1/2 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:58.836 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:07:58.979 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T16:07:58.979 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:58.980 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:07:58.980 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:07:58.980 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T16:07:58.980 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:07:58.980 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:07:59.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-volume 2024-03-04T16:07:59.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-03-04T16:07:59.716 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:32 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:07:59.890 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-1670.g1fb5d106.el9 ceph-noarch 264 k 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:07:59.892 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 332 k 2024-03-04T16:07:59.892 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.5 M 2024-03-04T16:07:59.892 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:07:59.915 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:33 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:00.084 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:00.085 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-04T16:08:00.085 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:00.085 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:08:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 127 k 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-03-04T16:08:00.086 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:08:00.143 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): ceph-volume-19.0.0-1670.g1fb5d106.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-04T16:08:00.167 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 250 kB/s | 69 kB 00:00 2024-03-04T16:08:00.168 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:00.169 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-04T16:08:00.169 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:00.178 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:00.178 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:00.227 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:00.227 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:00.380 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:00.556 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.0.0-1670.g1fb5d106.el9.x86_64 270 kB/s | 127 kB 00:00 2024-03-04T16:08:00.557 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:00.557 INFO:teuthology.orchestra.run.smithi071.stdout:Total 269 kB/s | 127 kB 00:00 2024-03-04T16:08:00.557 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:08:00.563 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:08:00.563 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:08:00.567 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T16:08:00.612 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2/2 2024-03-04T16:08:00.618 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:08:00.618 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:08:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2/2 2024-03-04T16:08:00.628 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:08:00.628 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T16:08:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:00.782 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:08:00.880 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:01.439 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 1/2 2024-03-04T16:08:01.662 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:01.662 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:01.662 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:08:01.663 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:01.663 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:01.663 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:01.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-1670.g1fb5d106.el9.noarch 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:01.789 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:01.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-03-04T16:08:02.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-03-04T16:08:02.476 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:02.591 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:02.646 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:02.648 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:02.648 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:02.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-03-04T16:08:02.730 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:35 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:02.905 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 127 k 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:08:02.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:02.907 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 127 k 2024-03-04T16:08:02.907 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 456 k 2024-03-04T16:08:02.907 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:08:03.134 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-19.0.0-1670.g1fb5d106.el9.x86_64 559 kB/s | 127 kB 00:00 2024-03-04T16:08:03.134 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:03.135 INFO:teuthology.orchestra.run.smithi027.stdout:Total 556 kB/s | 127 kB 00:00 2024-03-04T16:08:03.135 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:03.141 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:03.141 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:03.200 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:03.200 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:03.285 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:03.359 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:03.454 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:03.455 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:03.455 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 31 k 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 k 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 140 k 2024-03-04T16:08:03.456 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:08:03.471 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:03.778 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.0.0-1670.g1fb5d106.el9.x86_6 97 kB/s | 31 kB 00:00 2024-03-04T16:08:03.778 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:03.778 INFO:teuthology.orchestra.run.smithi071.stdout:Total 97 kB/s | 31 kB 00:00 2024-03-04T16:08:03.778 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:08:03.782 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:08:03.782 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:08:03.809 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:08:03.810 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:08:03.927 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:08:03.933 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:04.051 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:04.238 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:04.442 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:04.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:04.729 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:04.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-03-04T16:08:05.050 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:38 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:05.167 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:05.223 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:05.224 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:05.224 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:05.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-03-04T16:08:05.470 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:39 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:05.584 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:05.637 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:05.637 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:05.638 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:05.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-03-04T16:08:05.874 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:38 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:06.050 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:06.051 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-03-04T16:08:06.051 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:06.052 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:08:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 31 k 2024-03-04T16:08:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:06.052 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:08:06.052 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:06.053 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:08:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:06.053 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 31 k 2024-03-04T16:08:06.053 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 140 k 2024-03-04T16:08:06.053 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:08:06.249 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:40 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:06.296 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-19.0.0-1670.g1fb5d106.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-04T16:08:06.296 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:06.297 INFO:teuthology.orchestra.run.smithi027.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-04T16:08:06.297 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:06.301 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:06.301 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:06.345 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:06.345 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:06.364 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:06.415 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:06.416 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:06.416 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:06.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-03-04T16:08:06.462 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:07.022 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:07.136 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:07.189 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:07.189 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:07.189 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:07.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-03-04T16:08:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:07.250 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:07.250 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:07.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-03-04T16:08:07.811 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:07.924 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:07.975 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:07.976 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:07.976 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:08.014 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:08.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-03-04T16:08:08.133 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:08.189 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:08.190 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:08.190 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:08.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-03-04T16:08:08.590 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:08.706 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:08.758 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:08.759 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:08.759 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:08.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-03-04T16:08:08.830 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:08.948 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:09.004 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:09.005 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:09.005 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:09.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-03-04T16:08:09.402 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:09.524 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:09.579 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:09.580 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-04T16:08:09.580 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:09.632 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-03-04T16:08:09.653 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:09.770 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:09.825 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:09.826 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:09.826 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:09.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-03-04T16:08:10.201 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:44 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:10.369 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:10.369 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:10.369 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 85 k 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:08:10.370 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:10.371 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-03-04T16:08:10.371 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-03-04T16:08:10.371 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:08:10.478 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:43 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:10.596 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:10.650 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:10.651 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:10.652 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:10.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-03-04T16:08:10.747 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 227 kB/s | 85 kB 00:00 2024-03-04T16:08:10.747 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:10.748 INFO:teuthology.orchestra.run.smithi071.stdout:Total 226 kB/s | 85 kB 00:00 2024-03-04T16:08:10.749 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:08:10.755 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:08:10.755 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:08:10.810 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:08:10.811 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:08:10.959 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:08:11.053 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:11.301 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:44 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:11.420 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:11.476 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:11.477 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:11.477 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:11.505 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:11.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:11.817 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:12.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-03-04T16:08:12.134 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:45 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:12.253 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 is already installed. 2024-03-04T16:08:12.308 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:12.309 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-03-04T16:08:12.309 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:12.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-03-04T16:08:12.606 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:12.776 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:08:12.777 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:12.778 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.1 M 2024-03-04T16:08:12.778 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2024-03-04T16:08:12.778 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:08:12.957 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:45 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:13.133 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:13.134 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:13.134 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-04T16:08:13.134 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:13.134 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:08:13.134 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 85 k 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 85 k 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 231 k 2024-03-04T16:08:13.135 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:08:13.291 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 6.0 MB/s | 3.1 MB 00:00 2024-03-04T16:08:13.292 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:13.292 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.0 MB/s | 3.1 MB 00:00 2024-03-04T16:08:13.292 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:08:13.301 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:08:13.301 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:08:13.375 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:08:13.375 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:08:13.549 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 206 kB/s | 85 kB 00:00 2024-03-04T16:08:13.549 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:13.550 INFO:teuthology.orchestra.run.smithi027.stdout:Total 205 kB/s | 85 kB 00:00 2024-03-04T16:08:13.550 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:13.558 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:13.558 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:13.619 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:13.620 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:13.773 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:13.802 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:08:13.867 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:13.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T16:08:13.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:14.303 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:14.651 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:14.652 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:14.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-03-04T16:08:14.990 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:14.990 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:14.991 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:08:14.991 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:14.991 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:14.991 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:15.281 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-03-04T16:08:15.490 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:48 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 3.1 M 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:08:15.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:15.665 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.1 M 2024-03-04T16:08:15.665 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 13 M 2024-03-04T16:08:15.665 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:08:15.858 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:49 ago on Mon 04 Mar 2024 04:06:26 PM UTC. 2024-03-04T16:08:16.023 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 170 k 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 170 k 2024-03-04T16:08:16.025 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 490 k 2024-03-04T16:08:16.025 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-04T16:08:16.211 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-03-04T16:08:16.211 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:16.211 INFO:teuthology.orchestra.run.smithi027.stdout:Total 5.6 MB/s | 3.1 MB 00:00 2024-03-04T16:08:16.212 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:16.222 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:16.222 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:16.300 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:16.300 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:16.635 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 279 kB/s | 170 kB 00:00 2024-03-04T16:08:16.636 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:16.636 INFO:teuthology.orchestra.run.smithi071.stdout:Total 279 kB/s | 170 kB 00:00 2024-03-04T16:08:16.636 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-04T16:08:16.644 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-04T16:08:16.644 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-04T16:08:16.700 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-04T16:08:16.700 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-04T16:08:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:16.839 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T16:08:16.839 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T16:08:16.839 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T16:08:16.839 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T16:08:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:16.851 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-04T16:08:16.958 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:17.443 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:17.758 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-04T16:08:18.022 DEBUG:teuthology.parallel:result is None 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:18.023 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:18.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-03-04T16:08:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:51 ago on Mon 04 Mar 2024 04:06:27 PM UTC. 2024-03-04T16:08:19.026 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-04T16:08:19.027 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-04T16:08:19.027 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:19.027 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-1670.g1fb5d106.el9 ceph 170 k 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 170 k 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 490 k 2024-03-04T16:08:19.028 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-04T16:08:19.614 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-19.0.0-1670.g1fb5d106.el9.x86_64.rpm 291 kB/s | 170 kB 00:00 2024-03-04T16:08:19.615 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-04T16:08:19.615 INFO:teuthology.orchestra.run.smithi027.stdout:Total 290 kB/s | 170 kB 00:00 2024-03-04T16:08:19.615 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-04T16:08:19.623 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-04T16:08:19.624 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-04T16:08:19.683 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-04T16:08:19.683 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-04T16:08:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-04T16:08:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:20.404 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:20.709 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 1/1 2024-03-04T16:08:20.710 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:20.710 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-04T16:08:20.710 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-1670.g1fb5d106.el9.x86_64 2024-03-04T16:08:20.710 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:20.710 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-04T16:08:20.928 DEBUG:teuthology.parallel:result is None 2024-03-04T16:08:20.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-04T16:08:21.074 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T16:08:21.096 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-1670.g1fb5d106.el9 2024-03-04T16:08:21.096 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1670.g1fb5d106.el9 2024-03-04T16:08:21.097 INFO:teuthology.task.install:The correct ceph version 19.0.0-1670.g1fb5d106 is installed. 2024-03-04T16:08:21.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-03-04T16:08:21.240 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T16:08:21.265 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-1670.g1fb5d106.el9 2024-03-04T16:08:21.265 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1670.g1fb5d106.el9 2024-03-04T16:08:21.266 INFO:teuthology.task.install:The correct ceph version 19.0.0-1670.g1fb5d106 is installed. 2024-03-04T16:08:21.268 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T16:08:21.268 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:21.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T16:08:21.301 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:21.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T16:08:21.338 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T16:08:21.338 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:21.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T16:08:21.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T16:08:21.444 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:21.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T16:08:21.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T16:08:21.547 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T16:08:21.547 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:21.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T16:08:21.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T16:08:21.644 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:21.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T16:08:21.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T16:08:21.745 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T16:08:21.746 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:21.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T16:08:21.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T16:08:21.847 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:21.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T16:08:21.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T16:08:21.948 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T16:08:22.042 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-04T16:08:22.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-03-04T16:08:22.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-03-04T16:08:22.074 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-04T16:08:22.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T16:08:22.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T16:08:22.145 INFO:tasks.ceph:Creating extra log directories... 2024-03-04T16:08:22.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T16:08:22.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T16:08:22.225 INFO:tasks.ceph:Creating ceph cluster c1... 2024-03-04T16:08:22.225 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T16:08:22.225 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'branch': 'main', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7581264', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'main', 'sha1': 'a5335f5f0d4a33387826931c0d9be1a7ec43d970'}}, 'owner': 'scheduled_suriarte@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a5335f5f0d4a33387826931c0d9be1a7ec43d970', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_15:42:14', 'tube': 'smithi', 'user': 'suriarte', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T16:08:22.226 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-03-04T16:08:22.259 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-03-04T16:08:22.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T16:08:22.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T16:08:22.344 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:22.344 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T16:08:22.400 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T16:08:22.400 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-03-04T16:08:22.455 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T16:08:22.455 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-04 16:08:17.099456181 +0000 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-04 16:05:42.119743436 +0000 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-04 16:05:42.119743436 +0000 2024-03-04T16:08:22.456 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-04 16:05:42.119743436 +0000 2024-03-04T16:08:22.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T16:08:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-04T16:08:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-04T16:08:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000330062 s, 1.6 MB/s 2024-03-04T16:08:22.527 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T16:08:22.583 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-03-04T16:08:22.638 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T16:08:22.638 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:22.638 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-04 16:08:17.099456181 +0000 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-04 16:05:42.494733282 +0000 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-04 16:05:42.494733282 +0000 2024-03-04T16:08:22.639 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-04 16:05:42.494733282 +0000 2024-03-04T16:08:22.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T16:08:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-04T16:08:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-04T16:08:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000202116 s, 2.5 MB/s 2024-03-04T16:08:22.709 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T16:08:22.765 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-04 16:08:17.099456181 +0000 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-04 16:05:42.842723860 +0000 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-04 16:05:42.842723860 +0000 2024-03-04T16:08:22.820 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-04 16:05:42.842723860 +0000 2024-03-04T16:08:22.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T16:08:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-04T16:08:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-04T16:08:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000175595 s, 2.9 MB/s 2024-03-04T16:08:22.890 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T16:08:22.947 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-04 16:08:17.100456154 +0000 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-04 16:05:43.199714194 +0000 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-04 16:05:43.199714194 +0000 2024-03-04T16:08:23.002 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-04 16:05:43.199714194 +0000 2024-03-04T16:08:23.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T16:08:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-04T16:08:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-04T16:08:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000186453 s, 2.7 MB/s 2024-03-04T16:08:23.071 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T16:08:23.127 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:23.127 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T16:08:23.128 INFO:tasks.ceph:Generating config... 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] debug rgw = 20 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-03-04T16:08:23.129 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-03-04T16:08:23.130 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-03-04T16:08:23.130 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-03-04T16:08:23.130 INFO:tasks.ceph:[client] setgroup = ceph 2024-03-04T16:08:23.130 INFO:tasks.ceph:[client] setuser = ceph 2024-03-04T16:08:23.130 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-03-04T16:08:23.130 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T16:08:23.130 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T16:08:23.130 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T16:08:23.130 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T16:08:23.130 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T16:08:23.130 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T16:08:23.130 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T16:08:23.130 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-03-04T16:08:23.130 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-03-04T16:08:23.130 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:23.131 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:23.131 INFO:tasks.ceph:Setting up c1.mon.a... 2024-03-04T16:08:23.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-03-04T16:08:23.217 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/c1.keyring 2024-03-04T16:08:23.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-03-04T16:08:23.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T16:08:23.297 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.27')] 2024-03-04T16:08:23.297 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-03-04T16:08:23.298 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:23.298 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T16:08:23.352 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --print /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:08:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid ae90a905-a6e2-46ec-9c78-8b88d97a8526 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:fsid ae90a905-a6e2-46ec-9c78-8b88d97a8526 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-03-04T16:08:23.432153+0000 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-03-04T16:08:23.432153+0000 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-03-04T16:08:23.432 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-03-04T16:08:23.433 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-03-04T16:08:23.436 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T16:08:23.492 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID ae90a905-a6e2-46ec-9c78-8b88d97a8526... 2024-03-04T16:08:23.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-03-04T16:08:23.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.592 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.593 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-03-04T16:08:23.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.27 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = ae90a905-a6e2-46ec-9c78-8b88d97a8526 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-03-04T16:08:23.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-03-04T16:08:23.596 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-03-04T16:08:23.597 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-04T16:08:23.598 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw lifecycle = 20 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2024-03-04T16:08:23.600 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-04T16:08:23.601 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T16:08:23.602 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-04T16:08:23.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T16:08:23.604 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = 172.21.15.27 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ae90a905-a6e2-46ec-9c78-8b88d97a8526 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.605 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-04T16:08:23.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-04T16:08:23.607 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: rocksdb delete range threshold = 0 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-04T16:08:23.608 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.609 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T16:08:23.610 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw = 20 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw lifecycle = 20 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw sync = 20 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt require ssl = False 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt s3 kms backend = testing 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw curl low speed time = 300 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw data log num shards = 4 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw md log max shards = 4 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync data inject err probability = 0.1 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync log trim interval = 0 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync obj etag verify = True 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: setgroup = ceph 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout: setuser = ceph 2024-03-04T16:08:23.611 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout: rgw data notify interval msec = 0 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-03-04T16:08:23.612 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:23.613 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-03-04T16:08:23.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-03-04T16:08:23.673 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T16:08:23.673 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:23.673 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-03-04T16:08:23.688 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:23.688 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-03-04T16:08:23.743 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T16:08:23.743 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:23.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/c1.keyring 2024-03-04T16:08:23.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T16:08:23.829 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:23.830 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:23.885 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T16:08:23.885 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:23.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c1.keyring 2024-03-04T16:08:23.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-03-04T16:08:23.930 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:23.930 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:23.987 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T16:08:23.987 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T16:08:23.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-03-04T16:08:24.053 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-03-04T16:08:24.055 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T16:08:24.055 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T16:08:24.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-03-04T16:08:24.101 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/c1.client.0.keyring 2024-03-04T16:08:24.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-03-04T16:08:24.167 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/c1.client.1.keyring 2024-03-04T16:08:24.185 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T16:08:24.185 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T16:08:24.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-03-04T16:08:24.217 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:24.217 INFO:tasks.ceph:role: c1.osd.0 2024-03-04T16:08:24.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T16:08:24.217 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T16:08:24.294 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:24.295 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:24.802 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-04T16:08:24.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-04T16:08:24.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-03-04T16:08:24.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-03-04T16:08:24.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-03-04T16:08:25.067 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:25.067 INFO:tasks.ceph:role: c1.osd.1 2024-03-04T16:08:25.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T16:08:25.067 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T16:08:25.147 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:25.148 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:25.594 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-04T16:08:25.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-04T16:08:25.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-03-04T16:08:25.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-03-04T16:08:25.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-03-04T16:08:25.801 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:25.801 INFO:tasks.ceph:role: c1.osd.2 2024-03-04T16:08:25.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T16:08:25.802 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T16:08:25.879 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:25.880 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:26.362 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-04T16:08:26.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-04T16:08:26.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-03-04T16:08:26.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-03-04T16:08:26.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.652+0000 7f9b75c1f640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-03-04T16:08:26.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.653+0000 7f9b75c1f640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-03-04T16:08:26.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.653+0000 7f9b75c1f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T16:08:26.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.653+0000 7f9b75c1f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T16:08:26.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.653+0000 7f9b75c1f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-03-04T16:08:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:26.660+0000 7f9b75c1f640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-03-04T16:08:28.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-03-04T16:08:28.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:28.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.873+0000 7f2d9000c640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.873+0000 7f2d9000c640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.873+0000 7f2d9000c640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.873+0000 7f2d9000c640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.873+0000 7f2d9000c640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:28.874+0000 7f2d9000c640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-03-04T16:08:30.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-03-04T16:08:30.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.085+0000 7fd606a19640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-03-04T16:08:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.085+0000 7fd606a19640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-03-04T16:08:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.085+0000 7fd606a19640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T16:08:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.085+0000 7fd606a19640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T16:08:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.085+0000 7fd606a19640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-03-04T16:08:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:08:31.086+0000 7fd606a19640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-03-04T16:08:33.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-03-04T16:08:33.205 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T16:08:33.205 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-03-04T16:08:33.279 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-03-04T16:08:33.352 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-03-04T16:08:33.425 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-03-04T16:08:33.497 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.497 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-03-04T16:08:33.553 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:33.553 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-03-04T16:08:33.609 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T16:08:33.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/c1.keyring 2024-03-04T16:08:33.679 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-03-04T16:08:33.679 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD48eVlrSwrAxAAgK6ZwUyBJ0TKDidwmcCsmQ== 2024-03-04T16:08:33.679 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-03-04T16:08:33.679 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD68eVls8jsJhAAeSxSX/DTJ1/SiE8AzqPbxA== 2024-03-04T16:08:33.679 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD88eVlkykgNBAA9IjEZ0NHuv7bYBsLQZ8tXw== 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD/8eVlg0AjBRAAwL3aiC+XYPJk9cMo6D7O0w== 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD48eVlv00YBhAAiJNHFm9tyTQYwW5obn+5Zg== 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-03-04T16:08:33.680 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD48eVlVUj+CRAAP2/Bvb8r3PWylK9fh/RznA== 2024-03-04T16:08:33.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T16:08:33.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:08:33.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:08:33.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:08:33.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T16:08:34.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T16:08:34.065 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T16:08:34.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-03-04T16:08:34.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-03-04T16:08:34.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-03-04T16:08:34.228 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-03-04T16:08:34.284 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-03-04T16:08:34.284 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-03-04T16:08:34.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-03-04T16:08:34.327 INFO:tasks.ceph.c1.mon.a:Started 2024-03-04T16:08:34.327 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-03-04T16:08:34.327 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-03-04T16:08:34.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-03-04T16:08:34.329 INFO:tasks.ceph.c1.mgr.x:Started 2024-03-04T16:08:34.329 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T16:08:34.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd crush tunables default 2024-03-04T16:08:34.526 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-03-04T16:08:34.537 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T16:08:34.537 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-03-04T16:08:34.537 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:34.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-03-04T16:08:34.568 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:34.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-03-04T16:08:34.639 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:34.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-03-04T16:08:34.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd new 438d81fe-f7f1-4f98-84ae-6debbb3b20b7 0 2024-03-04T16:08:34.938 INFO:teuthology.orchestra.run.smithi027.stdout:0 2024-03-04T16:08:34.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd new 16f7de61-9fec-406f-bc8e-6857e1346a8a 1 2024-03-04T16:08:35.103 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.101+0000 7efcb9e4d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:08:35.140 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-04T16:08:35.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd new f4fdb92d-09a3-4e3b-868a-f57983224a2d 2 2024-03-04T16:08:35.159 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.159+0000 7efcb9e4d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:08:35.339 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-03-04T16:08:35.351 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-03-04T16:08:35.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-03-04T16:08:35.394 INFO:tasks.ceph.c1.osd.0:Started 2024-03-04T16:08:35.394 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-03-04T16:08:35.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-03-04T16:08:35.396 INFO:tasks.ceph.c1.osd.1:Started 2024-03-04T16:08:35.396 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-03-04T16:08:35.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-03-04T16:08:35.398 INFO:tasks.ceph.c1.osd.2:Started 2024-03-04T16:08:35.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T16:08:35.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:35.602 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"fsid":"ae90a905-a6e2-46ec-9c78-8b88d97a8526","created":"2024-03-04T16:08:34.446163+0000","modified":"2024-03-04T16:08:35.339543+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T16:08:35.339543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"438d81fe-f7f1-4f98-84ae-6debbb3b20b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"16f7de61-9fec-406f-bc8e-6857e1346a8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f4fdb92d-09a3-4e3b-868a-f57983224a2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:08:35.614 INFO:tasks.ceph.ceph_manager.c1:[] 2024-03-04T16:08:35.614 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T16:08:35.732 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.731+0000 7efcb9e4d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:08:35.803 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.802+0000 7efcb9e4d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:08:35.849 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.849+0000 7efcb9e4d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:08:35.906 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.905+0000 7efcb9e4d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:08:35.916 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-03-04T16:08:35.953 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:35.952+0000 7efcb9e4d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:08:36.001 INFO:tasks.ceph.c1.osd.0.smithi027.stderr:2024-03-04T16:08:36.000+0000 7f894bc13640 -1 Falling back to public interface 2024-03-04T16:08:36.022 INFO:tasks.ceph.c1.osd.2.smithi027.stderr:2024-03-04T16:08:36.021+0000 7f82df8a2640 -1 Falling back to public interface 2024-03-04T16:08:36.030 INFO:tasks.ceph.c1.osd.1.smithi027.stderr:2024-03-04T16:08:36.029+0000 7fe1791ff640 -1 Falling back to public interface 2024-03-04T16:08:36.083 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.082+0000 7efcb9e4d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:08:36.084 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-04T16:08:36.084 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":5,"fsid":"ae90a905-a6e2-46ec-9c78-8b88d97a8526","created":"2024-03-04T16:08:34.446163+0000","modified":"2024-03-04T16:08:35.339543+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T16:08:35.339543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"438d81fe-f7f1-4f98-84ae-6debbb3b20b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"16f7de61-9fec-406f-bc8e-6857e1346a8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f4fdb92d-09a3-4e3b-868a-f57983224a2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:08:36.094 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-04T16:08:36.157 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.156+0000 7efcb9e4d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:36.205 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.204+0000 7efcb9e4d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:08:36.260 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.259+0000 7efcb9e4d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:08:36.310 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.309+0000 7efcb9e4d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:08:36.712 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.711+0000 7efcb9e4d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:08:36.854 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.854+0000 7efcb9e4d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:08:36.983 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:36.982+0000 7efcb9e4d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:08:37.030 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.030+0000 7efcb9e4d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:08:37.148 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.147+0000 7efcb9e4d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:37.248 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.247+0000 7efcb9e4d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:08:37.409 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.409+0000 7efcb9e4d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:08:37.454 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.453+0000 7efcb9e4d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:08:37.616 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:37.615+0000 7efcb9e4d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:08:38.466 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:38.465+0000 7efcb9e4d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:08:38.719 INFO:tasks.ceph.c1.osd.0.smithi027.stderr:2024-03-04T16:08:38.718+0000 7f894bc13640 -1 osd.0 0 log_to_monitors true 2024-03-04T16:08:38.750 INFO:tasks.ceph.c1.osd.2.smithi027.stderr:2024-03-04T16:08:38.750+0000 7f82df8a2640 -1 osd.2 0 log_to_monitors true 2024-03-04T16:08:38.766 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:38.766+0000 7efcb9e4d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:08:38.771 INFO:tasks.ceph.c1.osd.1.smithi027.stderr:2024-03-04T16:08:38.771+0000 7fe1791ff640 -1 osd.1 0 log_to_monitors true 2024-03-04T16:08:38.822 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:38.821+0000 7efcb9e4d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:08:38.942 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:38.941+0000 7efcb9e4d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:08:40.015 INFO:tasks.ceph.c1.mgr.x.smithi027.stderr:2024-03-04T16:08:40.014+0000 7efc5aa37640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T16:08:40.658 INFO:tasks.ceph.c1.osd.0.smithi027.stderr:2024-03-04T16:08:40.657+0000 7f89475c0640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T16:08:40.668 INFO:tasks.ceph.c1.osd.2.smithi027.stderr:2024-03-04T16:08:40.667+0000 7f82db838640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T16:08:40.670 INFO:tasks.ceph.c1.osd.0.smithi027.stderr:2024-03-04T16:08:40.670+0000 7f893f3e0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:08:40.676 INFO:tasks.ceph.c1.osd.2.smithi027.stderr:2024-03-04T16:08:40.676+0000 7f82d3658640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:08:41.325 INFO:tasks.ceph.c1.osd.1.smithi027.stderr:2024-03-04T16:08:41.324+0000 7fe175195640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T16:08:41.334 INFO:tasks.ceph.c1.osd.1.smithi027.stderr:2024-03-04T16:08:41.334+0000 7fe16cfb5640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:08:42.398 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-03-04T16:08:42.697 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-04T16:08:42.697 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":9,"fsid":"ae90a905-a6e2-46ec-9c78-8b88d97a8526","created":"2024-03-04T16:08:34.446163+0000","modified":"2024-03-04T16:08:41.775104+0000","last_up_change":"2024-03-04T16:08:41.775104+0000","last_in_change":"2024-03-04T16:08:35.339543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:08:41.011438+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"438d81fe-f7f1-4f98-84ae-6debbb3b20b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6801","nonce":1997288047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6803","nonce":1997288047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6807","nonce":1997288047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6805","nonce":1997288047}]},"public_addr":"172.21.15.27:6801/1997288047","cluster_addr":"172.21.15.27:6803/1997288047","heartbeat_back_addr":"172.21.15.27:6807/1997288047","heartbeat_front_addr":"172.21.15.27:6805/1997288047","state":["exists","up"]},{"osd":1,"uuid":"16f7de61-9fec-406f-bc8e-6857e1346a8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6817","nonce":345040674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6819","nonce":345040674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6823","nonce":345040674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6821","nonce":345040674}]},"public_addr":"172.21.15.27:6817/345040674","cluster_addr":"172.21.15.27:6819/345040674","heartbeat_back_addr":"172.21.15.27:6823/345040674","heartbeat_front_addr":"172.21.15.27:6821/345040674","state":["exists","up"]},{"osd":2,"uuid":"f4fdb92d-09a3-4e3b-868a-f57983224a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6809","nonce":786433898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6811","nonce":786433898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6815","nonce":786433898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6813","nonce":786433898}]},"public_addr":"172.21.15.27:6809/786433898","cluster_addr":"172.21.15.27:6811/786433898","heartbeat_back_addr":"172.21.15.27:6815/786433898","heartbeat_front_addr":"172.21.15.27:6813/786433898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.769122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.737738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:08:42.708 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-04T16:08:42.708 INFO:tasks.ceph:Creating RBD pool 2024-03-04T16:08:42.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd pool create rbd 8 2024-03-04T16:08:43.783 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2024-03-04T16:08:43.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-04T16:08:44.784 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-04T16:08:44.802 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-03-04T16:08:44.803 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-03-04T16:08:44.803 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-03-04T16:08:44.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-03-04T16:08:44.804 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T16:08:45.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:45.167 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":2990170798},{"type":"v1","addr":"172.21.15.27:6825","nonce":2990170798}]},"active_addr":"172.21.15.27:6825/2990170798","active_change":"2024-03-04T16:08:38.955915+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1561657970}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3495478062}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4229372485}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1053698336}]}]} 2024-03-04T16:08:45.169 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-03-04T16:08:45.169 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-03-04T16:08:45.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T16:08:45.466 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:45.466 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"fsid":"ae90a905-a6e2-46ec-9c78-8b88d97a8526","created":"2024-03-04T16:08:34.446163+0000","modified":"2024-03-04T16:08:44.783422+0000","last_up_change":"2024-03-04T16:08:41.775104+0000","last_in_change":"2024-03-04T16:08:35.339543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:08:41.011438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T16:08:43.058233+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"438d81fe-f7f1-4f98-84ae-6debbb3b20b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6801","nonce":1997288047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6803","nonce":1997288047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6807","nonce":1997288047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6805","nonce":1997288047}]},"public_addr":"172.21.15.27:6801/1997288047","cluster_addr":"172.21.15.27:6803/1997288047","heartbeat_back_addr":"172.21.15.27:6807/1997288047","heartbeat_front_addr":"172.21.15.27:6805/1997288047","state":["exists","up"]},{"osd":1,"uuid":"16f7de61-9fec-406f-bc8e-6857e1346a8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6817","nonce":345040674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6819","nonce":345040674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6823","nonce":345040674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6821","nonce":345040674}]},"public_addr":"172.21.15.27:6817/345040674","cluster_addr":"172.21.15.27:6819/345040674","heartbeat_back_addr":"172.21.15.27:6823/345040674","heartbeat_front_addr":"172.21.15.27:6821/345040674","state":["exists","up"]},{"osd":2,"uuid":"f4fdb92d-09a3-4e3b-868a-f57983224a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6809","nonce":786433898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6811","nonce":786433898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6815","nonce":786433898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6813","nonce":786433898}]},"public_addr":"172.21.15.27:6809/786433898","cluster_addr":"172.21.15.27:6811/786433898","heartbeat_back_addr":"172.21.15.27:6815/786433898","heartbeat_front_addr":"172.21.15.27:6813/786433898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.769122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.812980+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.737738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:08:45.479 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-03-04T16:08:45.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-03-04T16:08:45.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:45.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"fsid":"ae90a905-a6e2-46ec-9c78-8b88d97a8526","created":"2024-03-04T16:08:34.446163+0000","modified":"2024-03-04T16:08:44.783422+0000","last_up_change":"2024-03-04T16:08:41.775104+0000","last_in_change":"2024-03-04T16:08:35.339543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:08:41.011438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T16:08:43.058233+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"438d81fe-f7f1-4f98-84ae-6debbb3b20b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6801","nonce":1997288047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6803","nonce":1997288047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6807","nonce":1997288047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1997288047},{"type":"v1","addr":"172.21.15.27:6805","nonce":1997288047}]},"public_addr":"172.21.15.27:6801/1997288047","cluster_addr":"172.21.15.27:6803/1997288047","heartbeat_back_addr":"172.21.15.27:6807/1997288047","heartbeat_front_addr":"172.21.15.27:6805/1997288047","state":["exists","up"]},{"osd":1,"uuid":"16f7de61-9fec-406f-bc8e-6857e1346a8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6817","nonce":345040674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6819","nonce":345040674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6823","nonce":345040674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":345040674},{"type":"v1","addr":"172.21.15.27:6821","nonce":345040674}]},"public_addr":"172.21.15.27:6817/345040674","cluster_addr":"172.21.15.27:6819/345040674","heartbeat_back_addr":"172.21.15.27:6823/345040674","heartbeat_front_addr":"172.21.15.27:6821/345040674","state":["exists","up"]},{"osd":2,"uuid":"f4fdb92d-09a3-4e3b-868a-f57983224a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6809","nonce":786433898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6811","nonce":786433898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6815","nonce":786433898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":786433898},{"type":"v1","addr":"172.21.15.27:6813","nonce":786433898}]},"public_addr":"172.21.15.27:6809/786433898","cluster_addr":"172.21.15.27:6811/786433898","heartbeat_back_addr":"172.21.15.27:6815/786433898","heartbeat_front_addr":"172.21.15.27:6813/786433898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.769122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.812980+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:08:39.737738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:08:45.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-03-04T16:08:45.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-03-04T16:08:45.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-03-04T16:08:45.936 INFO:teuthology.orchestra.run.smithi027.stdout:34359738371 2024-03-04T16:08:45.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-03-04T16:08:45.957 INFO:teuthology.orchestra.run.smithi027.stdout:34359738371 2024-03-04T16:08:45.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-03-04T16:08:45.958 INFO:teuthology.orchestra.run.smithi027.stdout:38654705668 2024-03-04T16:08:45.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-03-04T16:08:46.264 INFO:teuthology.orchestra.run.smithi027.stdout:34359738370 2024-03-04T16:08:46.280 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2024-03-04T16:08:46.316 INFO:teuthology.orchestra.run.smithi027.stdout:38654705667 2024-03-04T16:08:46.321 INFO:teuthology.orchestra.run.smithi027.stdout:34359738370 2024-03-04T16:08:46.329 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.1 2024-03-04T16:08:46.334 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2024-03-04T16:08:47.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-03-04T16:08:47.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-03-04T16:08:47.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-03-04T16:08:47.598 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2024-03-04T16:08:47.611 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.0 2024-03-04T16:08:47.612 DEBUG:teuthology.parallel:result is None 2024-03-04T16:08:47.642 INFO:teuthology.orchestra.run.smithi027.stdout:38654705668 2024-03-04T16:08:47.655 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.1 2024-03-04T16:08:47.655 DEBUG:teuthology.parallel:result is None 2024-03-04T16:08:47.682 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2024-03-04T16:08:47.695 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.2 2024-03-04T16:08:47.695 DEBUG:teuthology.parallel:result is None 2024-03-04T16:08:47.695 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-03-04T16:08:47.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-03-04T16:08:48.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-04T16:08:48.041 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-04T16:08:46.960987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82920,"kb_used_data":1472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489880,"statfs":{"total":322122547200,"available":322037637120,"internally_reserved":0,"allocated":1507328,"data_stored":1267331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.794125+0000","last_change":"2024-03-04T16:08:44.791452+0000","last_active":"2024-03-04T16:08:45.794125+0000","last_peered":"2024-03-04T16:08:45.794125+0000","last_clean":"2024-03-04T16:08:45.794125+0000","last_became_active":"2024-03-04T16:08:44.791144+0000","last_became_peered":"2024-03-04T16:08:44.791144+0000","last_unstale":"2024-03-04T16:08:45.794125+0000","last_undegraded":"2024-03-04T16:08:45.794125+0000","last_fullsized":"2024-03-04T16:08:45.794125+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:33:51.557756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793989+0000","last_change":"2024-03-04T16:08:44.792540+0000","last_active":"2024-03-04T16:08:45.793989+0000","last_peered":"2024-03-04T16:08:45.793989+0000","last_clean":"2024-03-04T16:08:45.793989+0000","last_became_active":"2024-03-04T16:08:44.792268+0000","last_became_peered":"2024-03-04T16:08:44.792268+0000","last_unstale":"2024-03-04T16:08:45.793989+0000","last_undegraded":"2024-03-04T16:08:45.793989+0000","last_fullsized":"2024-03-04T16:08:45.793989+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:26.812482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.794066+0000","last_change":"2024-03-04T16:08:44.795489+0000","last_active":"2024-03-04T16:08:45.794066+0000","last_peered":"2024-03-04T16:08:45.794066+0000","last_clean":"2024-03-04T16:08:45.794066+0000","last_became_active":"2024-03-04T16:08:44.795211+0000","last_became_peered":"2024-03-04T16:08:44.795211+0000","last_unstale":"2024-03-04T16:08:45.794066+0000","last_undegraded":"2024-03-04T16:08:45.794066+0000","last_fullsized":"2024-03-04T16:08:45.794066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:13.595227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793965+0000","last_change":"2024-03-04T16:08:44.792390+0000","last_active":"2024-03-04T16:08:45.793965+0000","last_peered":"2024-03-04T16:08:45.793965+0000","last_clean":"2024-03-04T16:08:45.793965+0000","last_became_active":"2024-03-04T16:08:44.791751+0000","last_became_peered":"2024-03-04T16:08:44.791751+0000","last_unstale":"2024-03-04T16:08:45.793965+0000","last_undegraded":"2024-03-04T16:08:45.793965+0000","last_fullsized":"2024-03-04T16:08:45.793965+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:57.865632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.798559+0000","last_change":"2024-03-04T16:08:44.795748+0000","last_active":"2024-03-04T16:08:44.798559+0000","last_peered":"2024-03-04T16:08:44.798559+0000","last_clean":"2024-03-04T16:08:44.798559+0000","last_became_active":"2024-03-04T16:08:44.795453+0000","last_became_peered":"2024-03-04T16:08:44.795453+0000","last_unstale":"2024-03-04T16:08:44.798559+0000","last_undegraded":"2024-03-04T16:08:44.798559+0000","last_fullsized":"2024-03-04T16:08:44.798559+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:26.234458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.796155+0000","last_change":"2024-03-04T16:08:44.791912+0000","last_active":"2024-03-04T16:08:44.796155+0000","last_peered":"2024-03-04T16:08:44.796155+0000","last_clean":"2024-03-04T16:08:44.796155+0000","last_became_active":"2024-03-04T16:08:44.791617+0000","last_became_peered":"2024-03-04T16:08:44.791617+0000","last_unstale":"2024-03-04T16:08:44.796155+0000","last_undegraded":"2024-03-04T16:08:44.796155+0000","last_fullsized":"2024-03-04T16:08:44.796155+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:08.922025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.796140+0000","last_change":"2024-03-04T16:08:44.791888+0000","last_active":"2024-03-04T16:08:44.796140+0000","last_peered":"2024-03-04T16:08:44.796140+0000","last_clean":"2024-03-04T16:08:44.796140+0000","last_became_active":"2024-03-04T16:08:44.791598+0000","last_became_peered":"2024-03-04T16:08:44.791598+0000","last_unstale":"2024-03-04T16:08:44.796140+0000","last_undegraded":"2024-03-04T16:08:44.796140+0000","last_fullsized":"2024-03-04T16:08:44.796140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:26.234458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793999+0000","last_change":"2024-03-04T16:08:44.792234+0000","last_active":"2024-03-04T16:08:45.793999+0000","last_peered":"2024-03-04T16:08:45.793999+0000","last_clean":"2024-03-04T16:08:45.793999+0000","last_became_active":"2024-03-04T16:08:44.791379+0000","last_became_peered":"2024-03-04T16:08:44.791379+0000","last_unstale":"2024-03-04T16:08:45.793999+0000","last_undegraded":"2024-03-04T16:08:45.793999+0000","last_fullsized":"2024-03-04T16:08:45.793999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:45:13.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793839+0000","last_change":"2024-03-04T16:08:42.785294+0000","last_active":"2024-03-04T16:08:45.793839+0000","last_peered":"2024-03-04T16:08:45.793839+0000","last_clean":"2024-03-04T16:08:45.793839+0000","last_became_active":"2024-03-04T16:08:42.784451+0000","last_became_peered":"2024-03-04T16:08:42.784451+0000","last_unstale":"2024-03-04T16:08:45.793839+0000","last_undegraded":"2024-03-04T16:08:45.793839+0000","last_fullsized":"2024-03-04T16:08:45.793839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:41.775104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:41.775104+0000","last_clean_scrub_stamp":"2024-03-04T16:08:41.775104+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:07.001373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27896,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829704,"statfs":{"total":107374182400,"available":107345616896,"internally_reserved":0,"allocated":700416,"data_stored":619233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27896,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829704,"statfs":{"total":107374182400,"available":107345616896,"internally_reserved":0,"allocated":700416,"data_stored":619233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:08:48.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-03-04T16:08:48.338 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:48.338 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-04T16:08:48.349 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-04T16:08:46.960987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82920,"kb_used_data":1472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489880,"statfs":{"total":322122547200,"available":322037637120,"internally_reserved":0,"allocated":1507328,"data_stored":1267331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.794125+0000","last_change":"2024-03-04T16:08:44.791452+0000","last_active":"2024-03-04T16:08:45.794125+0000","last_peered":"2024-03-04T16:08:45.794125+0000","last_clean":"2024-03-04T16:08:45.794125+0000","last_became_active":"2024-03-04T16:08:44.791144+0000","last_became_peered":"2024-03-04T16:08:44.791144+0000","last_unstale":"2024-03-04T16:08:45.794125+0000","last_undegraded":"2024-03-04T16:08:45.794125+0000","last_fullsized":"2024-03-04T16:08:45.794125+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:33:51.557756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793989+0000","last_change":"2024-03-04T16:08:44.792540+0000","last_active":"2024-03-04T16:08:45.793989+0000","last_peered":"2024-03-04T16:08:45.793989+0000","last_clean":"2024-03-04T16:08:45.793989+0000","last_became_active":"2024-03-04T16:08:44.792268+0000","last_became_peered":"2024-03-04T16:08:44.792268+0000","last_unstale":"2024-03-04T16:08:45.793989+0000","last_undegraded":"2024-03-04T16:08:45.793989+0000","last_fullsized":"2024-03-04T16:08:45.793989+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:26.812482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.794066+0000","last_change":"2024-03-04T16:08:44.795489+0000","last_active":"2024-03-04T16:08:45.794066+0000","last_peered":"2024-03-04T16:08:45.794066+0000","last_clean":"2024-03-04T16:08:45.794066+0000","last_became_active":"2024-03-04T16:08:44.795211+0000","last_became_peered":"2024-03-04T16:08:44.795211+0000","last_unstale":"2024-03-04T16:08:45.794066+0000","last_undegraded":"2024-03-04T16:08:45.794066+0000","last_fullsized":"2024-03-04T16:08:45.794066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:13.595227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793965+0000","last_change":"2024-03-04T16:08:44.792390+0000","last_active":"2024-03-04T16:08:45.793965+0000","last_peered":"2024-03-04T16:08:45.793965+0000","last_clean":"2024-03-04T16:08:45.793965+0000","last_became_active":"2024-03-04T16:08:44.791751+0000","last_became_peered":"2024-03-04T16:08:44.791751+0000","last_unstale":"2024-03-04T16:08:45.793965+0000","last_undegraded":"2024-03-04T16:08:45.793965+0000","last_fullsized":"2024-03-04T16:08:45.793965+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:57.865632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.798559+0000","last_change":"2024-03-04T16:08:44.795748+0000","last_active":"2024-03-04T16:08:44.798559+0000","last_peered":"2024-03-04T16:08:44.798559+0000","last_clean":"2024-03-04T16:08:44.798559+0000","last_became_active":"2024-03-04T16:08:44.795453+0000","last_became_peered":"2024-03-04T16:08:44.795453+0000","last_unstale":"2024-03-04T16:08:44.798559+0000","last_undegraded":"2024-03-04T16:08:44.798559+0000","last_fullsized":"2024-03-04T16:08:44.798559+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:26.234458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.796155+0000","last_change":"2024-03-04T16:08:44.791912+0000","last_active":"2024-03-04T16:08:44.796155+0000","last_peered":"2024-03-04T16:08:44.796155+0000","last_clean":"2024-03-04T16:08:44.796155+0000","last_became_active":"2024-03-04T16:08:44.791617+0000","last_became_peered":"2024-03-04T16:08:44.791617+0000","last_unstale":"2024-03-04T16:08:44.796155+0000","last_undegraded":"2024-03-04T16:08:44.796155+0000","last_fullsized":"2024-03-04T16:08:44.796155+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:08.922025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-04T16:08:44.796140+0000","last_change":"2024-03-04T16:08:44.791888+0000","last_active":"2024-03-04T16:08:44.796140+0000","last_peered":"2024-03-04T16:08:44.796140+0000","last_clean":"2024-03-04T16:08:44.796140+0000","last_became_active":"2024-03-04T16:08:44.791598+0000","last_became_peered":"2024-03-04T16:08:44.791598+0000","last_unstale":"2024-03-04T16:08:44.796140+0000","last_undegraded":"2024-03-04T16:08:44.796140+0000","last_fullsized":"2024-03-04T16:08:44.796140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:26.234458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793999+0000","last_change":"2024-03-04T16:08:44.792234+0000","last_active":"2024-03-04T16:08:45.793999+0000","last_peered":"2024-03-04T16:08:45.793999+0000","last_clean":"2024-03-04T16:08:45.793999+0000","last_became_active":"2024-03-04T16:08:44.791379+0000","last_became_peered":"2024-03-04T16:08:44.791379+0000","last_unstale":"2024-03-04T16:08:45.793999+0000","last_undegraded":"2024-03-04T16:08:45.793999+0000","last_fullsized":"2024-03-04T16:08:45.793999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:43.780475+0000","last_clean_scrub_stamp":"2024-03-04T16:08:43.780475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:45:13.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T16:08:45.793839+0000","last_change":"2024-03-04T16:08:42.785294+0000","last_active":"2024-03-04T16:08:45.793839+0000","last_peered":"2024-03-04T16:08:45.793839+0000","last_clean":"2024-03-04T16:08:45.793839+0000","last_became_active":"2024-03-04T16:08:42.784451+0000","last_became_peered":"2024-03-04T16:08:42.784451+0000","last_unstale":"2024-03-04T16:08:45.793839+0000","last_undegraded":"2024-03-04T16:08:45.793839+0000","last_fullsized":"2024-03-04T16:08:45.793839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:41.775104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:41.775104+0000","last_clean_scrub_stamp":"2024-03-04T16:08:41.775104+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:07.001373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27896,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829704,"statfs":{"total":107374182400,"available":107345616896,"internally_reserved":0,"allocated":700416,"data_stored":619233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27896,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829704,"statfs":{"total":107374182400,"available":107345616896,"internally_reserved":0,"allocated":700416,"data_stored":619233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:08:48.350 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-03-04T16:08:48.350 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-03-04T16:08:48.350 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-03-04T16:08:48.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-03-04T16:08:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:48.682 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T16:08:48.695 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-03-04T16:08:48.695 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T16:08:48.705 INFO:tasks.ceph:Creating ceph cluster c2... 2024-03-04T16:08:48.706 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T16:08:48.706 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'branch': 'main', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7581264', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'main', 'sha1': 'a5335f5f0d4a33387826931c0d9be1a7ec43d970'}}, 'owner': 'scheduled_suriarte@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a5335f5f0d4a33387826931c0d9be1a7ec43d970', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_15:42:14', 'tube': 'smithi', 'user': 'suriarte', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T16:08:48.706 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-03-04T16:08:48.709 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-03-04T16:08:48.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T16:08:48.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T16:08:48.796 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:48.797 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T16:08:48.854 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T16:08:48.854 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-04 16:08:14.130405778 +0000 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-04 16:05:34.754562005 +0000 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-04 16:05:34.754562005 +0000 2024-03-04T16:08:48.910 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-04 16:05:34.754562005 +0000 2024-03-04T16:08:48.910 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T16:08:48.977 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-04T16:08:48.977 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-04T16:08:48.977 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000202852 s, 2.5 MB/s 2024-03-04T16:08:48.978 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T16:08:49.035 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-04 16:08:14.131405761 +0000 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-04 16:05:35.090556280 +0000 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-04 16:05:35.090556280 +0000 2024-03-04T16:08:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-04 16:05:35.090556280 +0000 2024-03-04T16:08:49.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T16:08:49.158 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-04T16:08:49.158 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-04T16:08:49.158 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000226303 s, 2.3 MB/s 2024-03-04T16:08:49.159 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T16:08:49.215 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-04T16:08:49.270 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T16:08:49.270 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:49.270 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-04 16:08:14.131405761 +0000 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-04 16:05:35.438550350 +0000 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-04 16:05:35.438550350 +0000 2024-03-04T16:08:49.271 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-04 16:05:35.438550350 +0000 2024-03-04T16:08:49.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T16:08:49.336 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-04T16:08:49.336 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-04T16:08:49.336 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000172885 s, 3.0 MB/s 2024-03-04T16:08:49.337 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T16:08:49.393 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-04 16:08:14.132405744 +0000 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-04 16:05:35.960541459 +0000 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-04 16:05:35.960541459 +0000 2024-03-04T16:08:49.449 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-04 16:05:35.960541459 +0000 2024-03-04T16:08:49.450 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T16:08:49.518 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-04T16:08:49.518 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-04T16:08:49.518 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000180825 s, 2.8 MB/s 2024-03-04T16:08:49.520 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T16:08:49.577 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:49.577 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T16:08:49.577 INFO:tasks.ceph:Generating config... 2024-03-04T16:08:49.578 INFO:tasks.ceph:[client] debug rgw = 20 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] setgroup = ceph 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client] setuser = ceph 2024-03-04T16:08:49.579 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-03-04T16:08:49.580 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T16:08:49.580 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T16:08:49.580 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T16:08:49.580 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T16:08:49.580 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:49.580 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:49.580 INFO:tasks.ceph:Setting up c2.mon.a... 2024-03-04T16:08:49.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-03-04T16:08:49.672 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.keyring 2024-03-04T16:08:49.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-03-04T16:08:49.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T16:08:49.786 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.71')] 2024-03-04T16:08:49.787 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-03-04T16:08:49.788 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:49.788 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T16:08:49.841 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --print /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:49.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:08:49.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 810a80ac-1dd9-4234-b6c0-8090f60258dc 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = quincy 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 810a80ac-1dd9-4234-b6c0-8090f60258dc 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-03-04T16:08:49.920198+0000 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-03-04T16:08:49.920198+0000 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 17 (quincy) 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2024-03-04T16:08:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-03-04T16:08:49.921 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T16:08:49.976 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 810a80ac-1dd9-4234-b6c0-8090f60258dc... 2024-03-04T16:08:49.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-03-04T16:08:49.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-03-04T16:08:50.046 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-04T16:08:50.046 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-04T16:08:50.046 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T16:08:50.046 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-04T16:08:50.046 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-04T16:08:50.048 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-04T16:08:50.049 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = 172.21.15.71 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 810a80ac-1dd9-4234-b6c0-8090f60258dc 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-04T16:08:50.050 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-04T16:08:50.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: rocksdb delete range threshold = 0 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.052 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T16:08:50.053 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T16:08:50.054 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw = 20 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw lifecycle = 20 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: debug rgw sync = 20 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt require ssl = False 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt s3 kms backend = testing 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw curl low speed time = 300 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw data log num shards = 4 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw md log max shards = 4 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync data inject err probability = 0.1 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync log trim interval = 0 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw sync obj etag verify = True 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: setgroup = ceph 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: setuser = ceph 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-04T16:08:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: rgw data notify interval msec = 0 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-03-04T16:08:50.056 INFO:teuthology.orchestra.run.smithi027.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:50.066 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-03-04T16:08:50.068 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.71 2024-03-04T16:08:50.069 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 810a80ac-1dd9-4234-b6c0-8090f60258dc 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-03-04T16:08:50.070 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.071 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-03-04T16:08:50.072 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-04T16:08:50.073 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-03-04T16:08:50.074 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw lifecycle = 20 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2024-03-04T16:08:50.075 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-04T16:08:50.076 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2024-03-04T16:08:50.080 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-03-04T16:08:50.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-03-04T16:08:50.135 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T16:08:50.135 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:50.135 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-03-04T16:08:50.150 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:50.150 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-03-04T16:08:50.204 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-03-04T16:08:50.204 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:50.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/c2.keyring 2024-03-04T16:08:50.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T16:08:50.254 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-04T16:08:50.255 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:50.311 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T16:08:50.311 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:50.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c2.keyring 2024-03-04T16:08:50.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-03-04T16:08:50.351 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:50.351 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:50.406 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T16:08:50.406 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T16:08:50.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-03-04T16:08:50.501 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-03-04T16:08:50.503 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T16:08:50.503 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T16:08:50.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-03-04T16:08:50.545 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.client.0.keyring 2024-03-04T16:08:50.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-03-04T16:08:50.642 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.client.1.keyring 2024-03-04T16:08:50.657 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T16:08:50.657 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-04T16:08:50.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-03-04T16:08:50.724 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:50.724 INFO:tasks.ceph:role: c2.osd.0 2024-03-04T16:08:50.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T16:08:50.724 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T16:08:50.798 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:50.799 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:51.323 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-04T16:08:51.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-04T16:08:51.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-03-04T16:08:51.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-03-04T16:08:51.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-03-04T16:08:51.571 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:51.571 INFO:tasks.ceph:role: c2.osd.1 2024-03-04T16:08:51.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T16:08:51.571 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T16:08:51.645 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:51.645 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:51.646 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:52.102 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-04T16:08:52.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-04T16:08:52.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-03-04T16:08:52.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-03-04T16:08:52.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-03-04T16:08:52.295 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-04T16:08:52.295 INFO:tasks.ceph:role: c2.osd.2 2024-03-04T16:08:52.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-04T16:08:52.296 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T16:08:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T16:08:52.371 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T16:08:52.371 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T16:08:52.863 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-04T16:08:52.865 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-04T16:08:52.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-03-04T16:08:53.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-03-04T16:08:53.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:53.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.152+0000 7f23cca12640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-03-04T16:08:53.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.156+0000 7f23cca12640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-03-04T16:08:53.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.157+0000 7f23cca12640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T16:08:53.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.157+0000 7f23cca12640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T16:08:53.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.157+0000 7f23cca12640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-03-04T16:08:53.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:53.165+0000 7f23cca12640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-03-04T16:08:55.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-03-04T16:08:55.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.363+0000 7f147080d640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.363+0000 7f147080d640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.363+0000 7f147080d640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.363+0000 7f147080d640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.363+0000 7f147080d640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-03-04T16:08:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:55.364+0000 7f147080d640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-03-04T16:08:57.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-03-04T16:08:57.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.564+0000 7ff5c3611640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.565+0000 7ff5c3611640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.565+0000 7ff5c3611640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.565+0000 7ff5c3611640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.565+0000 7ff5c3611640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-03-04T16:08:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:08:57.565+0000 7ff5c3611640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-03-04T16:08:59.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-03-04T16:08:59.673 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T16:08:59.673 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:59.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-03-04T16:08:59.742 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:59.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-03-04T16:08:59.812 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:59.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-03-04T16:08:59.880 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:59.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-03-04T16:08:59.948 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:08:59.948 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-03-04T16:09:00.003 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:09:00.004 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-03-04T16:09:00.058 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T16:09:00.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/c2.keyring 2024-03-04T16:09:00.124 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-03-04T16:09:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAS8uVlmTzmHRAANJgJmUMGfiFBDTcc+CgAkw== 2024-03-04T16:09:00.124 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-04T16:09:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAV8uVlmJwsCRAAgoHAMIt8bTHuGrSMqGYGGQ== 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAX8uVlvZW3FRAAXAxcvVGaryMVTPMUqr5joA== 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAZ8uVl/++6IRAA9H07eVu9yMMqgkr2VooyPQ== 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAS8uVlFiaPIBAACFc+d1OXE3COMOEiY3FeeQ== 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-03-04T16:09:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAS8uVly6FUJhAAeAl4+vDJXzwSZxQW7EuEyg== 2024-03-04T16:09:00.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T16:09:00.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:09:00.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:09:00.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T16:09:00.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T16:09:00.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T16:09:00.485 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T16:09:00.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-03-04T16:09:00.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-03-04T16:09:00.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-03-04T16:09:00.635 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-03-04T16:09:00.690 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-03-04T16:09:00.690 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-03-04T16:09:00.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-03-04T16:09:00.733 INFO:tasks.ceph.c2.mon.a:Started 2024-03-04T16:09:00.733 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-03-04T16:09:00.733 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-03-04T16:09:00.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-03-04T16:09:00.735 INFO:tasks.ceph.c2.mgr.x:Started 2024-03-04T16:09:00.736 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T16:09:00.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd crush tunables default 2024-03-04T16:09:00.927 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-03-04T16:09:00.938 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T16:09:00.938 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-03-04T16:09:00.939 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:09:00.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-03-04T16:09:00.966 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:09:00.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-03-04T16:09:01.032 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-04T16:09:01.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-03-04T16:09:01.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new 1a850f78-b801-47de-b86c-8f0ec80b9cbf 0 2024-03-04T16:09:01.334 INFO:teuthology.orchestra.run.smithi071.stdout:0 2024-03-04T16:09:01.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new b2601ea8-8bd3-4728-ac8b-ebe9f16771e7 1 2024-03-04T16:09:01.497 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:01.496+0000 7f7591458040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:09:01.527 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-03-04T16:09:01.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new c06f75d5-f7b8-4571-9172-5fd3f2c47340 2 2024-03-04T16:09:01.553 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:01.552+0000 7f7591458040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:09:01.776 INFO:teuthology.orchestra.run.smithi071.stdout:2 2024-03-04T16:09:01.788 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-03-04T16:09:01.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-03-04T16:09:01.830 INFO:tasks.ceph.c2.osd.0:Started 2024-03-04T16:09:01.830 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-03-04T16:09:01.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-03-04T16:09:01.832 INFO:tasks.ceph.c2.osd.1:Started 2024-03-04T16:09:01.832 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-03-04T16:09:01.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-03-04T16:09:01.834 INFO:tasks.ceph.c2.osd.2:Started 2024-03-04T16:09:01.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T16:09:02.007 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:02.007 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"fsid":"810a80ac-1dd9-4234-b6c0-8090f60258dc","created":"2024-03-04T16:09:00.848567+0000","modified":"2024-03-04T16:09:01.776441+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T16:09:01.776441+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a850f78-b801-47de-b86c-8f0ec80b9cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2601ea8-8bd3-4728-ac8b-ebe9f16771e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c06f75d5-f7b8-4571-9172-5fd3f2c47340","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:09:02.018 INFO:tasks.ceph.ceph_manager.c2:[] 2024-03-04T16:09:02.018 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T16:09:02.236 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.235+0000 7f7591458040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:09:02.289 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.288+0000 7f7591458040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:09:02.320 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-03-04T16:09:02.335 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.335+0000 7f7591458040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:09:02.391 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.390+0000 7f7591458040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:09:02.430 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2024-03-04T16:09:02.430+0000 7f9523349640 -1 Falling back to public interface 2024-03-04T16:09:02.434 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2024-03-04T16:09:02.433+0000 7f2c06e0f640 -1 Falling back to public interface 2024-03-04T16:09:02.437 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.436+0000 7f7591458040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:09:02.450 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2024-03-04T16:09:02.450+0000 7fd8d9a13640 -1 Falling back to public interface 2024-03-04T16:09:02.483 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-04T16:09:02.483 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":5,"fsid":"810a80ac-1dd9-4234-b6c0-8090f60258dc","created":"2024-03-04T16:09:00.848567+0000","modified":"2024-03-04T16:09:01.776441+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T16:09:01.776441+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a850f78-b801-47de-b86c-8f0ec80b9cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2601ea8-8bd3-4728-ac8b-ebe9f16771e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c06f75d5-f7b8-4571-9172-5fd3f2c47340","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:09:02.493 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-04T16:09:02.559 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.558+0000 7f7591458040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:09:02.628 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.627+0000 7f7591458040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:09:02.673 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.672+0000 7f7591458040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:09:02.726 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.725+0000 7f7591458040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:09:02.773 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:02.773+0000 7f7591458040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:09:03.152 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.151+0000 7f7591458040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:09:03.286 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.286+0000 7f7591458040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:09:03.393 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.392+0000 7f7591458040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:09:03.437 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.436+0000 7f7591458040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:09:03.550 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.549+0000 7f7591458040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:09:03.645 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.644+0000 7f7591458040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:09:03.798 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.797+0000 7f7591458040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:09:03.841 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.840+0000 7f7591458040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:09:03.994 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:03.993+0000 7f7591458040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:09:05.097 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:05.096+0000 7f7591458040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:09:05.185 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2024-03-04T16:09:05.184+0000 7f2c06e0f640 -1 osd.1 0 log_to_monitors true 2024-03-04T16:09:05.190 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2024-03-04T16:09:05.189+0000 7f9523349640 -1 osd.0 0 log_to_monitors true 2024-03-04T16:09:05.232 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2024-03-04T16:09:05.231+0000 7fd8d9a13640 -1 osd.2 0 log_to_monitors true 2024-03-04T16:09:05.377 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:05.376+0000 7f7591458040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:09:05.428 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:05.428+0000 7f7591458040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:09:05.541 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:05.540+0000 7f7591458040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:09:06.597 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2024-03-04T16:09:06.596+0000 7f75328c2640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T16:09:07.528 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2024-03-04T16:09:07.528+0000 7f951f2df640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T16:09:07.547 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2024-03-04T16:09:07.546+0000 7fd8d5af9640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T16:09:07.551 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2024-03-04T16:09:07.551+0000 7f95170ff640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:09:07.564 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2024-03-04T16:09:07.564+0000 7fd8cd105640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:09:07.769 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2024-03-04T16:09:07.768+0000 7f2c026e5640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T16:09:07.778 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2024-03-04T16:09:07.778+0000 7f2bfa505640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:09:08.797 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-03-04T16:09:09.079 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-04T16:09:09.079 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":8,"fsid":"810a80ac-1dd9-4234-b6c0-8090f60258dc","created":"2024-03-04T16:09:00.848567+0000","modified":"2024-03-04T16:09:08.548745+0000","last_up_change":"2024-03-04T16:09:08.548745+0000","last_in_change":"2024-03-04T16:09:01.776441+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a850f78-b801-47de-b86c-8f0ec80b9cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6801","nonce":1751893534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6803","nonce":1751893534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6807","nonce":1751893534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6805","nonce":1751893534}]},"public_addr":"172.21.15.71:6801/1751893534","cluster_addr":"172.21.15.71:6803/1751893534","heartbeat_back_addr":"172.21.15.71:6807/1751893534","heartbeat_front_addr":"172.21.15.71:6805/1751893534","state":["exists","up"]},{"osd":1,"uuid":"b2601ea8-8bd3-4728-ac8b-ebe9f16771e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6809","nonce":1601354389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6811","nonce":1601354389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6815","nonce":1601354389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6813","nonce":1601354389}]},"public_addr":"172.21.15.71:6809/1601354389","cluster_addr":"172.21.15.71:6811/1601354389","heartbeat_back_addr":"172.21.15.71:6815/1601354389","heartbeat_front_addr":"172.21.15.71:6813/1601354389","state":["exists","up"]},{"osd":2,"uuid":"c06f75d5-f7b8-4571-9172-5fd3f2c47340","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6817","nonce":1278659300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6819","nonce":1278659300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6823","nonce":1278659300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6821","nonce":1278659300}]},"public_addr":"172.21.15.71:6817/1278659300","cluster_addr":"172.21.15.71:6819/1278659300","heartbeat_back_addr":"172.21.15.71:6823/1278659300","heartbeat_front_addr":"172.21.15.71:6821/1278659300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:09:09.090 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-04T16:09:09.090 INFO:tasks.ceph:Creating RBD pool 2024-03-04T16:09:09.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd pool create rbd 8 2024-03-04T16:09:09.608 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2024-03-04T16:09:09.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-04T16:09:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-04T16:09:10.633 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-03-04T16:09:10.633 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-03-04T16:09:10.633 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-03-04T16:09:10.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-03-04T16:09:10.634 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T16:09:10.996 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:11.007 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3627361616},{"type":"v1","addr":"172.21.15.71:6825","nonce":3627361616}]},"active_addr":"172.21.15.71:6825/3627361616","active_change":"2024-03-04T16:09:05.555177+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":144135792}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2490523325}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2663772682}]}]} 2024-03-04T16:09:11.012 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-03-04T16:09:11.012 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-03-04T16:09:11.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:11.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"810a80ac-1dd9-4234-b6c0-8090f60258dc","created":"2024-03-04T16:09:00.848567+0000","modified":"2024-03-04T16:09:10.608647+0000","last_up_change":"2024-03-04T16:09:08.548745+0000","last_in_change":"2024-03-04T16:09:01.776441+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T16:09:09.387814+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T16:09:09.602427+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a850f78-b801-47de-b86c-8f0ec80b9cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6801","nonce":1751893534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6803","nonce":1751893534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6807","nonce":1751893534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6805","nonce":1751893534}]},"public_addr":"172.21.15.71:6801/1751893534","cluster_addr":"172.21.15.71:6803/1751893534","heartbeat_back_addr":"172.21.15.71:6807/1751893534","heartbeat_front_addr":"172.21.15.71:6805/1751893534","state":["exists","up"]},{"osd":1,"uuid":"b2601ea8-8bd3-4728-ac8b-ebe9f16771e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6809","nonce":1601354389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6811","nonce":1601354389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6815","nonce":1601354389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6813","nonce":1601354389}]},"public_addr":"172.21.15.71:6809/1601354389","cluster_addr":"172.21.15.71:6811/1601354389","heartbeat_back_addr":"172.21.15.71:6815/1601354389","heartbeat_front_addr":"172.21.15.71:6813/1601354389","state":["exists","up"]},{"osd":2,"uuid":"c06f75d5-f7b8-4571-9172-5fd3f2c47340","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6817","nonce":1278659300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6819","nonce":1278659300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6823","nonce":1278659300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6821","nonce":1278659300}]},"public_addr":"172.21.15.71:6817/1278659300","cluster_addr":"172.21.15.71:6819/1278659300","heartbeat_back_addr":"172.21.15.71:6823/1278659300","heartbeat_front_addr":"172.21.15.71:6821/1278659300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:06.165885+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:06.157719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:07.548132+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:09:11.342 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-03-04T16:09:11.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-03-04T16:09:11.624 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:11.625 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":11,"fsid":"810a80ac-1dd9-4234-b6c0-8090f60258dc","created":"2024-03-04T16:09:00.848567+0000","modified":"2024-03-04T16:09:11.612157+0000","last_up_change":"2024-03-04T16:09:08.548745+0000","last_in_change":"2024-03-04T16:09:01.776441+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-03-04T16:09:09.387814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-03-04T16:09:09.602427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a850f78-b801-47de-b86c-8f0ec80b9cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6801","nonce":1751893534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6803","nonce":1751893534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6807","nonce":1751893534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1751893534},{"type":"v1","addr":"172.21.15.71:6805","nonce":1751893534}]},"public_addr":"172.21.15.71:6801/1751893534","cluster_addr":"172.21.15.71:6803/1751893534","heartbeat_back_addr":"172.21.15.71:6807/1751893534","heartbeat_front_addr":"172.21.15.71:6805/1751893534","state":["exists","up"]},{"osd":1,"uuid":"b2601ea8-8bd3-4728-ac8b-ebe9f16771e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6809","nonce":1601354389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6811","nonce":1601354389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6815","nonce":1601354389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1601354389},{"type":"v1","addr":"172.21.15.71:6813","nonce":1601354389}]},"public_addr":"172.21.15.71:6809/1601354389","cluster_addr":"172.21.15.71:6811/1601354389","heartbeat_back_addr":"172.21.15.71:6815/1601354389","heartbeat_front_addr":"172.21.15.71:6813/1601354389","state":["exists","up"]},{"osd":2,"uuid":"c06f75d5-f7b8-4571-9172-5fd3f2c47340","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6817","nonce":1278659300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6819","nonce":1278659300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6823","nonce":1278659300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":1278659300},{"type":"v1","addr":"172.21.15.71:6821","nonce":1278659300}]},"public_addr":"172.21.15.71:6817/1278659300","cluster_addr":"172.21.15.71:6819/1278659300","heartbeat_back_addr":"172.21.15.71:6823/1278659300","heartbeat_front_addr":"172.21.15.71:6821/1278659300","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:06.165885+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:06.157719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:09:07.548132+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:09:11.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-03-04T16:09:11.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-03-04T16:09:11.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-03-04T16:09:11.776 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:11.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T16:09:11.780 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:11.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T16:09:11.782 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:11.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T16:09:12.096 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:12.109 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2024-03-04T16:09:12.135 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:12.147 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2024-03-04T16:09:12.152 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:12.165 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2024-03-04T16:09:13.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T16:09:13.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T16:09:13.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T16:09:13.418 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:13.431 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2024-03-04T16:09:13.455 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:13.467 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2024-03-04T16:09:13.497 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2024-03-04T16:09:13.510 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2024-03-04T16:09:14.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-03-04T16:09:14.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-03-04T16:09:14.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-03-04T16:09:14.747 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:14.759 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2024-03-04T16:09:14.760 DEBUG:teuthology.parallel:result is None 2024-03-04T16:09:14.785 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:14.798 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2024-03-04T16:09:14.798 DEBUG:teuthology.parallel:result is None 2024-03-04T16:09:14.822 INFO:teuthology.orchestra.run.smithi071.stdout:34359738371 2024-03-04T16:09:14.835 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2024-03-04T16:09:14.835 DEBUG:teuthology.parallel:result is None 2024-03-04T16:09:14.835 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-03-04T16:09:14.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-03-04T16:09:15.153 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:15.153 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-03-04T16:09:15.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-04T16:09:13.562426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82732,"kb_used_data":1424,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490068,"statfs":{"total":322122547200,"available":322037829632,"internally_reserved":0,"allocated":1458176,"data_stored":1248584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616809+0000","last_change":"2024-03-04T16:09:10.623541+0000","last_active":"2024-03-04T16:09:11.616809+0000","last_peered":"2024-03-04T16:09:11.616809+0000","last_clean":"2024-03-04T16:09:11.616809+0000","last_became_active":"2024-03-04T16:09:10.623228+0000","last_became_peered":"2024-03-04T16:09:10.623228+0000","last_unstale":"2024-03-04T16:09:11.616809+0000","last_undegraded":"2024-03-04T16:09:11.616809+0000","last_fullsized":"2024-03-04T16:09:11.616809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:30:20.195905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616719+0000","last_change":"2024-03-04T16:09:10.618912+0000","last_active":"2024-03-04T16:09:11.616719+0000","last_peered":"2024-03-04T16:09:11.616719+0000","last_clean":"2024-03-04T16:09:11.616719+0000","last_became_active":"2024-03-04T16:09:10.618625+0000","last_became_peered":"2024-03-04T16:09:10.618625+0000","last_unstale":"2024-03-04T16:09:11.616719+0000","last_undegraded":"2024-03-04T16:09:11.616719+0000","last_fullsized":"2024-03-04T16:09:11.616719+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:04.568959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.625635+0000","last_change":"2024-03-04T16:09:10.620536+0000","last_active":"2024-03-04T16:09:10.625635+0000","last_peered":"2024-03-04T16:09:10.625635+0000","last_clean":"2024-03-04T16:09:10.625635+0000","last_became_active":"2024-03-04T16:09:10.620098+0000","last_became_peered":"2024-03-04T16:09:10.620098+0000","last_unstale":"2024-03-04T16:09:10.625635+0000","last_undegraded":"2024-03-04T16:09:10.625635+0000","last_fullsized":"2024-03-04T16:09:10.625635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:39.870371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616774+0000","last_change":"2024-03-04T16:09:10.619260+0000","last_active":"2024-03-04T16:09:11.616774+0000","last_peered":"2024-03-04T16:09:11.616774+0000","last_clean":"2024-03-04T16:09:11.616774+0000","last_became_active":"2024-03-04T16:09:10.618981+0000","last_became_peered":"2024-03-04T16:09:10.618981+0000","last_unstale":"2024-03-04T16:09:11.616774+0000","last_undegraded":"2024-03-04T16:09:11.616774+0000","last_fullsized":"2024-03-04T16:09:11.616774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:27:22.422004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.165188+0000","last_change":"2024-03-04T16:09:10.618174+0000","last_active":"2024-03-04T16:09:11.165188+0000","last_peered":"2024-03-04T16:09:11.165188+0000","last_clean":"2024-03-04T16:09:11.165188+0000","last_became_active":"2024-03-04T16:09:10.617861+0000","last_became_peered":"2024-03-04T16:09:10.617861+0000","last_unstale":"2024-03-04T16:09:11.165188+0000","last_undegraded":"2024-03-04T16:09:11.165188+0000","last_fullsized":"2024-03-04T16:09:11.165188+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:17:28.137512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616705+0000","last_change":"2024-03-04T16:09:10.621361+0000","last_active":"2024-03-04T16:09:11.616705+0000","last_peered":"2024-03-04T16:09:11.616705+0000","last_clean":"2024-03-04T16:09:11.616705+0000","last_became_active":"2024-03-04T16:09:10.621097+0000","last_became_peered":"2024-03-04T16:09:10.621097+0000","last_unstale":"2024-03-04T16:09:11.616705+0000","last_undegraded":"2024-03-04T16:09:11.616705+0000","last_fullsized":"2024-03-04T16:09:11.616705+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:23.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.620441+0000","last_change":"2024-03-04T16:09:10.618199+0000","last_active":"2024-03-04T16:09:10.620441+0000","last_peered":"2024-03-04T16:09:10.620441+0000","last_clean":"2024-03-04T16:09:10.620441+0000","last_became_active":"2024-03-04T16:09:10.617893+0000","last_became_peered":"2024-03-04T16:09:10.617893+0000","last_unstale":"2024-03-04T16:09:10.620441+0000","last_undegraded":"2024-03-04T16:09:10.620441+0000","last_fullsized":"2024-03-04T16:09:10.620441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:17:28.137512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.625650+0000","last_change":"2024-03-04T16:09:10.620547+0000","last_active":"2024-03-04T16:09:10.625650+0000","last_peered":"2024-03-04T16:09:10.625650+0000","last_clean":"2024-03-04T16:09:10.625650+0000","last_became_active":"2024-03-04T16:09:10.620220+0000","last_became_peered":"2024-03-04T16:09:10.620220+0000","last_unstale":"2024-03-04T16:09:10.625650+0000","last_undegraded":"2024-03-04T16:09:10.625650+0000","last_fullsized":"2024-03-04T16:09:10.625650+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:36.879873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616829+0000","last_change":"2024-03-04T16:09:10.618131+0000","last_active":"2024-03-04T16:09:11.616829+0000","last_peered":"2024-03-04T16:09:11.616829+0000","last_clean":"2024-03-04T16:09:11.616829+0000","last_became_active":"2024-03-04T16:09:10.617869+0000","last_became_peered":"2024-03-04T16:09:10.617869+0000","last_unstale":"2024-03-04T16:09:11.616829+0000","last_undegraded":"2024-03-04T16:09:11.616829+0000","last_fullsized":"2024-03-04T16:09:11.616829+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:19.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27940,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829660,"statfs":{"total":107374182400,"available":107345571840,"internally_reserved":0,"allocated":684032,"data_stored":612984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":684032,"data_stored":612984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27108,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":90112,"data_stored":22616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:09:15.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-03-04T16:09:15.442 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:15.443 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-03-04T16:09:15.455 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-04T16:09:13.562426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82732,"kb_used_data":1424,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490068,"statfs":{"total":322122547200,"available":322037829632,"internally_reserved":0,"allocated":1458176,"data_stored":1248584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616809+0000","last_change":"2024-03-04T16:09:10.623541+0000","last_active":"2024-03-04T16:09:11.616809+0000","last_peered":"2024-03-04T16:09:11.616809+0000","last_clean":"2024-03-04T16:09:11.616809+0000","last_became_active":"2024-03-04T16:09:10.623228+0000","last_became_peered":"2024-03-04T16:09:10.623228+0000","last_unstale":"2024-03-04T16:09:11.616809+0000","last_undegraded":"2024-03-04T16:09:11.616809+0000","last_fullsized":"2024-03-04T16:09:11.616809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:30:20.195905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616719+0000","last_change":"2024-03-04T16:09:10.618912+0000","last_active":"2024-03-04T16:09:11.616719+0000","last_peered":"2024-03-04T16:09:11.616719+0000","last_clean":"2024-03-04T16:09:11.616719+0000","last_became_active":"2024-03-04T16:09:10.618625+0000","last_became_peered":"2024-03-04T16:09:10.618625+0000","last_unstale":"2024-03-04T16:09:11.616719+0000","last_undegraded":"2024-03-04T16:09:11.616719+0000","last_fullsized":"2024-03-04T16:09:11.616719+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:04.568959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.625635+0000","last_change":"2024-03-04T16:09:10.620536+0000","last_active":"2024-03-04T16:09:10.625635+0000","last_peered":"2024-03-04T16:09:10.625635+0000","last_clean":"2024-03-04T16:09:10.625635+0000","last_became_active":"2024-03-04T16:09:10.620098+0000","last_became_peered":"2024-03-04T16:09:10.620098+0000","last_unstale":"2024-03-04T16:09:10.625635+0000","last_undegraded":"2024-03-04T16:09:10.625635+0000","last_fullsized":"2024-03-04T16:09:10.625635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:39.870371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616774+0000","last_change":"2024-03-04T16:09:10.619260+0000","last_active":"2024-03-04T16:09:11.616774+0000","last_peered":"2024-03-04T16:09:11.616774+0000","last_clean":"2024-03-04T16:09:11.616774+0000","last_became_active":"2024-03-04T16:09:10.618981+0000","last_became_peered":"2024-03-04T16:09:10.618981+0000","last_unstale":"2024-03-04T16:09:11.616774+0000","last_undegraded":"2024-03-04T16:09:11.616774+0000","last_fullsized":"2024-03-04T16:09:11.616774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:27:22.422004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.165188+0000","last_change":"2024-03-04T16:09:10.618174+0000","last_active":"2024-03-04T16:09:11.165188+0000","last_peered":"2024-03-04T16:09:11.165188+0000","last_clean":"2024-03-04T16:09:11.165188+0000","last_became_active":"2024-03-04T16:09:10.617861+0000","last_became_peered":"2024-03-04T16:09:10.617861+0000","last_unstale":"2024-03-04T16:09:11.165188+0000","last_undegraded":"2024-03-04T16:09:11.165188+0000","last_fullsized":"2024-03-04T16:09:11.165188+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:17:28.137512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616705+0000","last_change":"2024-03-04T16:09:10.621361+0000","last_active":"2024-03-04T16:09:11.616705+0000","last_peered":"2024-03-04T16:09:11.616705+0000","last_clean":"2024-03-04T16:09:11.616705+0000","last_became_active":"2024-03-04T16:09:10.621097+0000","last_became_peered":"2024-03-04T16:09:10.621097+0000","last_unstale":"2024-03-04T16:09:11.616705+0000","last_undegraded":"2024-03-04T16:09:11.616705+0000","last_fullsized":"2024-03-04T16:09:11.616705+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:23.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.620441+0000","last_change":"2024-03-04T16:09:10.618199+0000","last_active":"2024-03-04T16:09:10.620441+0000","last_peered":"2024-03-04T16:09:10.620441+0000","last_clean":"2024-03-04T16:09:10.620441+0000","last_became_active":"2024-03-04T16:09:10.617893+0000","last_became_peered":"2024-03-04T16:09:10.617893+0000","last_unstale":"2024-03-04T16:09:10.620441+0000","last_undegraded":"2024-03-04T16:09:10.620441+0000","last_fullsized":"2024-03-04T16:09:10.620441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:17:28.137512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.625650+0000","last_change":"2024-03-04T16:09:10.620547+0000","last_active":"2024-03-04T16:09:10.625650+0000","last_peered":"2024-03-04T16:09:10.625650+0000","last_clean":"2024-03-04T16:09:10.625650+0000","last_became_active":"2024-03-04T16:09:10.620220+0000","last_became_peered":"2024-03-04T16:09:10.620220+0000","last_unstale":"2024-03-04T16:09:10.625650+0000","last_undegraded":"2024-03-04T16:09:10.625650+0000","last_fullsized":"2024-03-04T16:09:10.625650+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:36.879873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.616829+0000","last_change":"2024-03-04T16:09:10.618131+0000","last_active":"2024-03-04T16:09:11.616829+0000","last_peered":"2024-03-04T16:09:11.616829+0000","last_clean":"2024-03-04T16:09:11.616829+0000","last_became_active":"2024-03-04T16:09:10.617869+0000","last_became_peered":"2024-03-04T16:09:10.617869+0000","last_unstale":"2024-03-04T16:09:11.616829+0000","last_undegraded":"2024-03-04T16:09:11.616829+0000","last_fullsized":"2024-03-04T16:09:11.616829+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:09.602760+0000","last_clean_scrub_stamp":"2024-03-04T16:09:09.602760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:19.007033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27940,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829660,"statfs":{"total":107374182400,"available":107345571840,"internally_reserved":0,"allocated":684032,"data_stored":612984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":684032,"data_stored":612984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27108,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830492,"statfs":{"total":107374182400,"available":107346423808,"internally_reserved":0,"allocated":90112,"data_stored":22616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:09:15.456 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-03-04T16:09:15.456 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-03-04T16:09:15.456 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-03-04T16:09:15.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-03-04T16:09:15.773 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-04T16:09:15.773 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T16:09:15.785 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-03-04T16:09:15.786 INFO:teuthology.run_tasks:Running task rgw... 2024-03-04T16:09:15.800 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-03-04T16:09:15.801 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-03-04T16:09:15.801 INFO:tasks.rgw:Creating data pools 2024-03-04T16:09:15.801 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-03-04T16:09:15.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-03-04T16:09:17.098 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'default.rgw.buckets.data' created 2024-03-04T16:09:17.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-03-04T16:09:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T16:09:18.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-03-04T16:09:19.106 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'default.rgw.buckets.index' created 2024-03-04T16:09:19.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-03-04T16:09:20.107 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T16:09:20.143 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-03-04T16:09:20.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-03-04T16:09:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'default.rgw.buckets.data' already exists 2024-03-04T16:09:20.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-03-04T16:09:21.116 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T16:09:21.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-03-04T16:09:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'default.rgw.buckets.index' already exists 2024-03-04T16:09:21.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-03-04T16:09:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T16:09:22.133 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-03-04T16:09:22.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-03-04T16:09:22.636 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' created 2024-03-04T16:09:22.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-03-04T16:09:23.638 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T16:09:23.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-03-04T16:09:24.642 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' created 2024-03-04T16:09:24.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-03-04T16:09:25.648 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T16:09:25.673 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-03-04T16:09:25.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-03-04T16:09:25.961 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' already exists 2024-03-04T16:09:25.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-03-04T16:09:26.650 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-04T16:09:26.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-03-04T16:09:26.942 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' already exists 2024-03-04T16:09:26.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-03-04T16:09:27.653 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-04T16:09:27.667 DEBUG:tasks.rgw:Pools created 2024-03-04T16:09:27.668 INFO:tasks.rgw:Configuring compression type = random 2024-03-04T16:09:27.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-03-04T16:09:27.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-03-04T16:09:27.668 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-03-04T16:09:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.187+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 realm 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.188+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.189+0000 7fcfeca2e800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:29.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.189+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.190+0000 7fcfeca2e800 20 rados_obj.operate() r=0 bl.length=875 2024-03-04T16:09:29.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.190+0000 7fcfeca2e800 20 searching for the correct realm 2024-03-04T16:09:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got zone_info.f2e0e5fd-55e1-411a-a2eb-714d91a33b04 2024-03-04T16:09:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got zonegroup_info.3ee30913-25d3-4a2a-9c80-0ad10df58788 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.199+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 zone default found 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 4 Realm: () 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 4 ZoneGroup: default (3ee30913-25d3-4a2a-9c80-0ad10df58788) 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 4 Zone: default (f2e0e5fd-55e1-411a-a2eb-714d91a33b04) 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T16:09:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 zonegroup default 2024-03-04T16:09:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:29.200+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:31.173+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:31.173+0000 7fcfeca2e800 20 rados->read ofs=0 len=0 2024-03-04T16:09:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:31.175+0000 7fcfeca2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:31.175+0000 7fcfeca2e800 20 started sync module instance, tier type = 2024-03-04T16:09:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:31.175+0000 7fcfeca2e800 20 started zone id=f2e0e5fd-55e1-411a-a2eb-714d91a33b04 (name=default) with tier type = 2024-03-04T16:09:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=0 2024-03-04T16:09:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=1 2024-03-04T16:09:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=2 2024-03-04T16:09:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=3 2024-03-04T16:09:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=4 2024-03-04T16:09:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=5 2024-03-04T16:09:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.196+0000 7fcfeca2e800 20 add_watcher() i=6 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.197+0000 7fcfeca2e800 20 add_watcher() i=7 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.197+0000 7fcfeca2e800 2 all 8 watchers are set, enabling cache 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-03-04T16:09:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.198+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.199+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-03-04T16:09:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.199+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-03-04T16:09:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.199+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.199+0000 7fcfeca2e800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-03-04T16:09:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.199+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-03-04T16:09:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.200+0000 7fcfeca2e800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.203+0000 7fce99fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.203+0000 7fcfeca2e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:33.203+0000 7fcfeca2e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.176+0000 7fcfeca2e800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-03-04T16:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.176+0000 7fcfeca2e800 5 note: GC not initialized 2024-03-04T16:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.176+0000 7fce98fe1640 20 reqs_thread_entry: start 2024-03-04T16:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.176+0000 7fcfeca2e800 20 init_complete bucket index max shards: 11 2024-03-04T16:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.176+0000 7fcfeca2e800 20 Filter name: none 2024-03-04T16:09:36.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.177+0000 7fce027fc640 20 reqs_thread_entry: start 2024-03-04T16:09:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.200+0000 7fce99fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.203+0000 7fcfeca2e800 20 remove_watcher() i=0 2024-03-04T16:09:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.203+0000 7fcfeca2e800 2 removed watcher, disabling cache 2024-03-04T16:09:36.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.205+0000 7fcfeca2e800 20 remove_watcher() i=1 2024-03-04T16:09:36.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.208+0000 7fcfeca2e800 20 remove_watcher() i=2 2024-03-04T16:09:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.210+0000 7fcfeca2e800 20 remove_watcher() i=3 2024-03-04T16:09:36.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.213+0000 7fcfeca2e800 20 remove_watcher() i=4 2024-03-04T16:09:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.215+0000 7fcfeca2e800 20 remove_watcher() i=5 2024-03-04T16:09:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.218+0000 7fcfeca2e800 20 remove_watcher() i=6 2024-03-04T16:09:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.221+0000 7fcfeca2e800 20 remove_watcher() i=7 2024-03-04T16:09:36.234 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-03-04T16:09:36.234 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T16:09:36.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:36.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:36.235 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T16:09:36.285 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:36.285 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.308+0000 7fc353011800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.308+0000 7fc353011800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.311+0000 7fc19effd640 20 reqs_thread_entry: start 2024-03-04T16:09:36.326 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"f2e0e5fd-55e1-411a-a2eb-714d91a33b04","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T16:09:36.326 DEBUG:tasks.util.rgw: json result: {'id': 'f2e0e5fd-55e1-411a-a2eb-714d91a33b04', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T16:09:36.327 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-03-04T16:09:36.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-03-04T16:09:36.327 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-03-04T16:09:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.397+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 realm 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.398+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.399+0000 7f5a9880f800 20 rados_obj.operate() r=0 bl.length=891 2024-03-04T16:09:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.399+0000 7f5a9880f800 20 searching for the correct realm 2024-03-04T16:09:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got zone_info.f2e0e5fd-55e1-411a-a2eb-714d91a33b04 2024-03-04T16:09:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T16:09:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T16:09:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got zonegroup_info.3ee30913-25d3-4a2a-9c80-0ad10df58788 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.406+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 zone default found 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 4 Realm: () 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 4 ZoneGroup: default (3ee30913-25d3-4a2a-9c80-0ad10df58788) 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 4 Zone: default (f2e0e5fd-55e1-411a-a2eb-714d91a33b04) 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 zonegroup default 2024-03-04T16:09:36.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.407+0000 7f5a9880f800 20 rados->read ofs=0 len=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.408+0000 7f5a9880f800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.408+0000 7f5a9880f800 20 started sync module instance, tier type = 2024-03-04T16:09:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.408+0000 7f5a9880f800 20 started zone id=f2e0e5fd-55e1-411a-a2eb-714d91a33b04 (name=default) with tier type = 2024-03-04T16:09:36.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.427+0000 7f5a9880f800 20 add_watcher() i=0 2024-03-04T16:09:36.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=1 2024-03-04T16:09:36.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=2 2024-03-04T16:09:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=3 2024-03-04T16:09:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=4 2024-03-04T16:09:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=5 2024-03-04T16:09:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.428+0000 7f5a9880f800 20 add_watcher() i=6 2024-03-04T16:09:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.430+0000 7f5a9880f800 20 add_watcher() i=7 2024-03-04T16:09:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.430+0000 7f5a9880f800 2 all 8 watchers are set, enabling cache 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5949fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5a9880f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5a9880f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5a9880f800 5 note: GC not initialized 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5948fe1640 20 reqs_thread_entry: start 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5a9880f800 20 init_complete bucket index max shards: 11 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f5a9880f800 20 Filter name: none 2024-03-04T16:09:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.433+0000 7f58c27fc640 20 reqs_thread_entry: start 2024-03-04T16:09:36.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.441+0000 7f5949fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:36.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.445+0000 7f5a9880f800 20 remove_watcher() i=0 2024-03-04T16:09:36.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.445+0000 7f5a9880f800 2 removed watcher, disabling cache 2024-03-04T16:09:36.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.445+0000 7f5a9880f800 20 remove_watcher() i=1 2024-03-04T16:09:36.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.446+0000 7f5a9880f800 20 remove_watcher() i=2 2024-03-04T16:09:36.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.447+0000 7f5a9880f800 20 remove_watcher() i=3 2024-03-04T16:09:36.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.448+0000 7f5a9880f800 20 remove_watcher() i=4 2024-03-04T16:09:36.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.449+0000 7f5a9880f800 20 remove_watcher() i=5 2024-03-04T16:09:36.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.450+0000 7f5a9880f800 20 remove_watcher() i=6 2024-03-04T16:09:36.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.451+0000 7f5a9880f800 20 remove_watcher() i=7 2024-03-04T16:09:36.461 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-03-04T16:09:36.461 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T16:09:36.461 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:36.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:36.462 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T16:09:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.574+0000 7fb8f969d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.574+0000 7fb8f969d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:36.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:36.576+0000 7fb7b5ffb640 20 reqs_thread_entry: start 2024-03-04T16:09:36.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"f2e0e5fd-55e1-411a-a2eb-714d91a33b04","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T16:09:36.592 DEBUG:tasks.util.rgw: json result: {'id': 'f2e0e5fd-55e1-411a-a2eb-714d91a33b04', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T16:09:36.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-03-04T16:09:36.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-03-04T16:09:36.592 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-03-04T16:09:36.699 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:36.699 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:38.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.720+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 realm 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:38.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.721+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.722+0000 7fe73934b800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:38.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.722+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.722+0000 7fe73934b800 20 rados_obj.operate() r=0 bl.length=875 2024-03-04T16:09:38.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.722+0000 7fe73934b800 20 searching for the correct realm 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got zonegroup_info.8468baa8-8882-4cd0-a2b8-d949d7f413ef 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got zone_info.12f96b64-5704-4208-827d-bd16668305c4 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T16:09:38.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.730+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:38.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 zone default found 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 4 Realm: () 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 4 ZoneGroup: default (8468baa8-8882-4cd0-a2b8-d949d7f413ef) 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 4 Zone: default (12f96b64-5704-4208-827d-bd16668305c4) 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 zonegroup default 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.731+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.732+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:38.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:38.732+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:40.697+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:40.697+0000 7fe73934b800 20 rados->read ofs=0 len=0 2024-03-04T16:09:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:40.698+0000 7fe73934b800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:40.698+0000 7fe73934b800 20 started sync module instance, tier type = 2024-03-04T16:09:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:40.698+0000 7fe73934b800 20 started zone id=12f96b64-5704-4208-827d-bd16668305c4 (name=default) with tier type = 2024-03-04T16:09:42.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=0 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=1 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=2 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=3 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=4 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=5 2024-03-04T16:09:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.746+0000 7fe73934b800 20 add_watcher() i=6 2024-03-04T16:09:42.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.748+0000 7fe73934b800 20 add_watcher() i=7 2024-03-04T16:09:42.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.748+0000 7fe73934b800 2 all 8 watchers are set, enabling cache 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-03-04T16:09:42.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.749+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.750+0000 7fe73934b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-03-04T16:09:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.750+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-03-04T16:09:42.752 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.751+0000 7fe73934b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-04T16:09:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.756+0000 7fe5e97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.756+0000 7fe73934b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:42.756+0000 7fe73934b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:45.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.707+0000 7fe73934b800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-03-04T16:09:45.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.707+0000 7fe73934b800 5 note: GC not initialized 2024-03-04T16:09:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.708+0000 7fe553fff640 20 reqs_thread_entry: start 2024-03-04T16:09:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.708+0000 7fe73934b800 20 init_complete bucket index max shards: 11 2024-03-04T16:09:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.708+0000 7fe73934b800 20 Filter name: none 2024-03-04T16:09:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.708+0000 7fe551ffb640 20 reqs_thread_entry: start 2024-03-04T16:09:45.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.730+0000 7fe5e97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:45.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.732+0000 7fe73934b800 20 remove_watcher() i=0 2024-03-04T16:09:45.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.732+0000 7fe73934b800 2 removed watcher, disabling cache 2024-03-04T16:09:45.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.734+0000 7fe73934b800 20 remove_watcher() i=1 2024-03-04T16:09:45.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.736+0000 7fe73934b800 20 remove_watcher() i=2 2024-03-04T16:09:45.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.741+0000 7fe73934b800 20 remove_watcher() i=3 2024-03-04T16:09:45.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.742+0000 7fe73934b800 20 remove_watcher() i=4 2024-03-04T16:09:45.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.744+0000 7fe73934b800 20 remove_watcher() i=5 2024-03-04T16:09:45.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.745+0000 7fe73934b800 20 remove_watcher() i=6 2024-03-04T16:09:45.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.748+0000 7fe73934b800 20 remove_watcher() i=7 2024-03-04T16:09:45.775 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2024-03-04T16:09:45.775 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T16:09:45.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:45.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:45.776 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T16:09:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.848+0000 7f0b9a733800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.848+0000 7f0b9a733800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:45.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.850+0000 7f0a58fe1640 20 reqs_thread_entry: start 2024-03-04T16:09:45.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"12f96b64-5704-4208-827d-bd16668305c4","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T16:09:45.865 DEBUG:tasks.util.rgw: json result: {'id': '12f96b64-5704-4208-827d-bd16668305c4', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T16:09:45.865 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-03-04T16:09:45.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-03-04T16:09:45.865 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-03-04T16:09:45.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:45.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:45.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.932+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.932+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.932+0000 7feb2fd33800 20 realm 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.932+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:45.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 rados_obj.operate() r=0 bl.length=891 2024-03-04T16:09:45.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.933+0000 7feb2fd33800 20 searching for the correct realm 2024-03-04T16:09:45.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got zonegroup_info.8468baa8-8882-4cd0-a2b8-d949d7f413ef 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got zone_info.12f96b64-5704-4208-827d-bd16668305c4 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got default.zone. 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got zone_names.default 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 rados_obj.operate() r=0 bl.length=46 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.941+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados_obj.operate() r=0 bl.length=398 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 zone default found 2024-03-04T16:09:45.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 4 Realm: () 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 4 ZoneGroup: default (8468baa8-8882-4cd0-a2b8-d949d7f413ef) 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 4 Zone: default (12f96b64-5704-4208-827d-bd16668305c4) 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 zonegroup default 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.942+0000 7feb2fd33800 20 rados->read ofs=0 len=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.943+0000 7feb2fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-03-04T16:09:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.943+0000 7feb2fd33800 20 started sync module instance, tier type = 2024-03-04T16:09:45.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.943+0000 7feb2fd33800 20 started zone id=12f96b64-5704-4208-827d-bd16668305c4 (name=default) with tier type = 2024-03-04T16:09:45.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=0 2024-03-04T16:09:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=1 2024-03-04T16:09:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=2 2024-03-04T16:09:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=3 2024-03-04T16:09:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=4 2024-03-04T16:09:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=5 2024-03-04T16:09:45.955 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=6 2024-03-04T16:09:45.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.952+0000 7feb2fd33800 20 add_watcher() i=7 2024-03-04T16:09:45.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.953+0000 7feb2fd33800 2 all 8 watchers are set, enabling cache 2024-03-04T16:09:45.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7fe9e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:45.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7feb2fd33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7feb2fd33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7feb2fd33800 5 note: GC not initialized 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7fe9e0fe1640 20 reqs_thread_entry: start 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7feb2fd33800 20 init_complete bucket index max shards: 11 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.954+0000 7feb2fd33800 20 Filter name: none 2024-03-04T16:09:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.955+0000 7fe95a7fc640 20 reqs_thread_entry: start 2024-03-04T16:09:45.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.962+0000 7fe9e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-04T16:09:45.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.964+0000 7feb2fd33800 20 remove_watcher() i=0 2024-03-04T16:09:45.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.964+0000 7feb2fd33800 2 removed watcher, disabling cache 2024-03-04T16:09:45.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.965+0000 7feb2fd33800 20 remove_watcher() i=1 2024-03-04T16:09:45.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.966+0000 7feb2fd33800 20 remove_watcher() i=2 2024-03-04T16:09:45.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.967+0000 7feb2fd33800 20 remove_watcher() i=3 2024-03-04T16:09:45.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.968+0000 7feb2fd33800 20 remove_watcher() i=4 2024-03-04T16:09:45.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.969+0000 7feb2fd33800 20 remove_watcher() i=5 2024-03-04T16:09:45.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.970+0000 7feb2fd33800 20 remove_watcher() i=6 2024-03-04T16:09:45.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:45.971+0000 7feb2fd33800 20 remove_watcher() i=7 2024-03-04T16:09:45.981 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2024-03-04T16:09:45.981 DEBUG:tasks.util.rgw: json result: [] 2024-03-04T16:09:45.981 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:45.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-04T16:09:45.982 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-04T16:09:46.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:46.069 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:46.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:46.090+0000 7fa1f0b33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-04T16:09:46.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:46.090+0000 7fa1f0b33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-04T16:09:46.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:09:46.092+0000 7fa0327fc640 20 reqs_thread_entry: start 2024-03-04T16:09:46.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"12f96b64-5704-4208-827d-bd16668305c4","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2024-03-04T16:09:46.107 DEBUG:tasks.util.rgw: json result: {'id': '12f96b64-5704-4208-827d-bd16668305c4', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-03-04T16:09:46.107 INFO:tasks.rgw:Starting rgw... 2024-03-04T16:09:46.107 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-03-04T16:09:46.108 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T16:09:46.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo echo -n http://smithi027.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.146 INFO:teuthology.orchestra.run.smithi027.stdout:http://smithi027.front.sepia.ceph.com:8000 2024-03-04T16:09:46.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.223 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-03-04T16:09:46.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-03-04T16:09:46.265 INFO:tasks.rgw.c1.client.0:Started 2024-03-04T16:09:46.265 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-03-04T16:09:46.265 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T16:09:46.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo echo -n http://smithi027.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.301 INFO:teuthology.orchestra.run.smithi027.stdout:http://smithi027.front.sepia.ceph.com:8001 2024-03-04T16:09:46.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.375 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-03-04T16:09:46.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-03-04T16:09:46.418 INFO:tasks.rgw.c1.client.1:Started 2024-03-04T16:09:46.418 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-03-04T16:09:46.418 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T16:09:46.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo echo -n http://smithi071.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.447 INFO:teuthology.orchestra.run.smithi071.stdout:http://smithi071.front.sepia.ceph.com:8000 2024-03-04T16:09:46.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.515 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-03-04T16:09:46.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-03-04T16:09:46.557 INFO:tasks.rgw.c2.client.0:Started 2024-03-04T16:09:46.557 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-03-04T16:09:46.557 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-04T16:09:46.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo echo -n http://smithi071.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.585 INFO:teuthology.orchestra.run.smithi071.stdout:http://smithi071.front.sepia.ceph.com:8001 2024-03-04T16:09:46.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-04T16:09:46.653 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-03-04T16:09:46.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-03-04T16:09:46.696 INFO:tasks.rgw.c2.client.1:Started 2024-03-04T16:09:46.696 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi027.front.sepia.ceph.com:8000/ 2024-03-04T16:09:46.696 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8000/ 2024-03-04T16:09:46.730 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:09:46.731 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:09:46.732 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T16:09:46.733 INFO:teuthology.orchestra.run.smithi027.stdout:anonymous 2024-03-04T16:09:46.734 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi027.front.sepia.ceph.com:8001/ 2024-03-04T16:09:46.734 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8001/ 2024-03-04T16:09:46.797 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:09:46.797 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:09:46.800 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T16:09:46.800 INFO:teuthology.orchestra.run.smithi027.stdout:anonymous 2024-03-04T16:09:46.800 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8000/ 2024-03-04T16:09:46.800 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000/ 2024-03-04T16:09:46.830 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:09:46.830 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:09:46.833 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-03-04T16:09:46.833 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2024-03-04T16:09:46.833 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8001/ 2024-03-04T16:09:46.833 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001/ 2024-03-04T16:09:46.894 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:09:46.894 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:09:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T16:09:46.898 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T16:09:46.898 INFO:teuthology.orchestra.run.smithi071.stderr:curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8001: Connection refused 2024-03-04T16:09:47.899 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001/ 2024-03-04T16:09:47.918 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:09:47.918 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:09:47.919 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-03-04T16:09:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2024-03-04T16:09:47.920 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-03-04T16:09:48.797 INFO:tasks.rgw_multisite:creating realm.. 2024-03-04T16:09:48.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.798 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:09:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:48.899 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"2fe35df2-135e-4762-bdce-6c2b1b264663","name":"test-realm","current_period":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","epoch":1} 2024-03-04T16:09:48.899 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-03-04T16:09:48.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.900 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:09:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:48.973+0000 7f70cb53c800 -1 old format 2024-03-04T16:09:48.988 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-03-04T16:09:48.988 INFO:tasks.rgw_multisite:creating master zone.. 2024-03-04T16:09:48.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi027.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi027.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:48.989 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key K5KCDZD1BZNDFSI4 --secret bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm --endpoints http://smithi027.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:09:49.082 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:49.111 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: set zone's realm_id=2fe35df2-135e-4762-bdce-6c2b1b264663 2024-03-04T16:09:49.133 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone1.rgw.log:notif"} 2024-03-04T16:09:49.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-03-04T16:09:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-03-04T16:09:50.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-03-04T16:09:51.211 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-03-04T16:09:51.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:51.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:51.246 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:09:51.299 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:51.299 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:51.340 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone1.rgw.log:notif"} 2024-03-04T16:09:51.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:51.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:09:51.340 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:09:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:09:51.419+0000 7f0c53a44800 0 period (11c84af7-2aba-4dd8-b74e-932e4b0eb09a does not have zone fc294220-5038-4f25-aeee-cac2c5838729 configured 2024-03-04T16:09:58.323 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":1,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T16:09:58.324 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-03-04T16:09:58.324 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T16:09:58.325 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T16:09:58.324+0000 7ff0615da640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 39044) UID: 0 2024-03-04T16:09:58.325 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T16:09:58.324+0000 7ff064e33880 -1 shutting down 2024-03-04T16:10:02.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T16:10:04.425 INFO:tasks.rgw.c1.client.0:Stopped 2024-03-04T16:10:04.425 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-03-04T16:10:04.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-03-04T16:10:04.428 INFO:tasks.rgw.c1.client.0:Started 2024-03-04T16:10:04.429 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8000 2024-03-04T16:10:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T16:10:04.451 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:04.452 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:04.452 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T16:10:04.452 INFO:teuthology.orchestra.run.smithi027.stderr:curl: (7) Failed to connect to smithi027.front.sepia.ceph.com port 8000: Connection refused 2024-03-04T16:10:05.453 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8000 2024-03-04T16:10:05.474 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:05.474 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:05.476 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-03-04T16:10:05.476 INFO:teuthology.orchestra.run.smithi027.stdout:anonymous 2024-03-04T16:10:05.476 INFO:tasks.rgw_multisite:creating admin user.. 2024-03-04T16:10:05.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.477 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key K5KCDZD1BZNDFSI4 --secret bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:05.572 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:05.573 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:05.620 INFO:teuthology.orchestra.run.smithi027.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm","active":true}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-03-04T16:10:05.638 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-03-04T16:10:05.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi027.front.sepia.ceph.com:8000', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi027.front.sepia.ceph.com:8000', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.638 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi027.front.sepia.ceph.com:8000 --access-key K5KCDZD1BZNDFSI4 --secret bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:05.695 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:05.695 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:05.768 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"2fe35df2-135e-4762-bdce-6c2b1b264663","name":"test-realm","current_period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":2} 2024-03-04T16:10:05.768 INFO:tasks.rgw_multisite:creating zone.. 2024-03-04T16:10:05.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:05.769 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key K5KCDZD1BZNDFSI4 --secret bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm --endpoints http://smithi071.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:05.820 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:05.820 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:05.847 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=2fe35df2-135e-4762-bdce-6c2b1b264663 2024-03-04T16:10:05.868 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T16:10:05.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-03-04T16:10:06.727 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-03-04T16:10:06.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-03-04T16:10:07.732 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-03-04T16:10:07.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:07.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:07.765 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:07.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:07.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:07.855 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T16:10:07.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:07.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:07.856 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:07.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:07.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:07.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:07.929+0000 7f51e871f800 0 period (0c9eafce-04d6-4013-9950-f1b02bed5b7e does not have zone a1482fc7-8de0-4f7d-8e05-b02ecec7d664 configured 2024-03-04T16:10:08.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-03-04T16:10:14.796 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone fc294220-5038-4f25-aeee-cac2c5838729 2024-03-04T16:10:14.844 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":2,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T16:10:14.844 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-03-04T16:10:14.844 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T16:10:14.845 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2024-03-04T16:10:14.845+0000 7fd7a56a8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 39313) UID: 0 2024-03-04T16:10:14.845 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2024-03-04T16:10:14.845+0000 7fd7a8f01880 -1 shutting down 2024-03-04T16:10:18.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T16:10:20.946 INFO:tasks.rgw.c2.client.0:Stopped 2024-03-04T16:10:20.946 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-03-04T16:10:20.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-03-04T16:10:20.949 INFO:tasks.rgw.c2.client.0:Started 2024-03-04T16:10:20.949 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2024-03-04T16:10:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T16:10:20.971 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:20.971 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:20.971 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T16:10:20.971 INFO:teuthology.orchestra.run.smithi071.stderr:curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8000: Connection refused 2024-03-04T16:10:21.972 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2024-03-04T16:10:21.992 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:21.992 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:21.993 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-03-04T16:10:21.993 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2024-03-04T16:10:21.994 INFO:tasks.rgw_multisite:creating zone.. 2024-03-04T16:10:21.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi027.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:21.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K5KCDZD1BZNDFSI4', '--secret', 'bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm', '--endpoints', 'http://smithi027.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:21.994 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key K5KCDZD1BZNDFSI4 --secret bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm --endpoints http://smithi027.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: set zone's realm_id=2fe35df2-135e-4762-bdce-6c2b1b264663 2024-03-04T16:10:22.111 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone3.rgw.log:notif"} 2024-03-04T16:10:22.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-03-04T16:10:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-03-04T16:10:23.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-03-04T16:10:24.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-03-04T16:10:24.277 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-03-04T16:10:24.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:24.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:24.304 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:24.366 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:24.366 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:24.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone3.rgw.log:notif"} 2024-03-04T16:10:24.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:24.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:24.407 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:24.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:10:24.485+0000 7fd39ba3a800 0 period (0c9eafce-04d6-4013-9950-f1b02bed5b7e does not have zone dce1012f-1cf8-44d9-b80b-aa1da8d03699 configured 2024-03-04T16:10:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:Sending period to new master zone fc294220-5038-4f25-aeee-cac2c5838729 2024-03-04T16:10:31.405 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":3,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","endpoints":["http://smithi027.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","val":210978822},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T16:10:31.406 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-03-04T16:10:31.406 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T16:10:31.406 INFO:tasks.rgw.c1.client.1.smithi027.stdout:2024-03-04T16:10:31.406+0000 7efc8e9d6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 39676) UID: 0 2024-03-04T16:10:31.407 INFO:tasks.rgw.c1.client.1.smithi027.stdout:2024-03-04T16:10:31.406+0000 7efc9222f880 -1 shutting down 2024-03-04T16:10:37.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T16:10:37.508 INFO:tasks.rgw.c1.client.1:Stopped 2024-03-04T16:10:37.509 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-03-04T16:10:37.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-03-04T16:10:37.512 INFO:tasks.rgw.c1.client.1:Started 2024-03-04T16:10:37.512 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8001 2024-03-04T16:10:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-04T16:10:37.536 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:37.536 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:37.536 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-04T16:10:37.536 INFO:teuthology.orchestra.run.smithi027.stderr:curl: (7) Failed to connect to smithi027.front.sepia.ceph.com port 8001: Connection refused 2024-03-04T16:10:38.537 DEBUG:teuthology.orchestra.run.smithi027:> curl http://smithi027.front.sepia.ceph.com:8001 2024-03-04T16:10:38.558 INFO:teuthology.orchestra.run.smithi027.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-04T16:10:38.558 INFO:teuthology.orchestra.run.smithi027.stderr: Dload Upload Total Spent Left Speed 2024-03-04T16:10:38.559 INFO:teuthology.orchestra.run.smithi027.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-03-04T16:10:38.560 INFO:teuthology.orchestra.run.smithi027.stdout:anonymous 2024-03-04T16:10:38.560 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-03-04T16:10:38.560 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-03-04T16:10:38.575 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi 2024-03-04T16:10:38.606 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-03-04T16:10:38.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:38.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:10:38.607 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T16:10:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:38.714 INFO:teuthology.orchestra.run.smithi027.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"4LUC3XJ67HIPQ3U1NKIY","secret_key":"GKwDjUJZ54ehrI2pM0bd9VR2VdTLHnXOfrzfRnEa","active":true}],"swift_keys":[],"caps":[{"type":"roles","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2024-03-04T16:10:38.734 INFO:rgw_multi.tests:meta checkpoint 2024-03-04T16:10:38.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:38.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:38.735 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:38.826 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:10:38.845 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:10:38.845 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:10:38.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:38.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:38.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:38.946+0000 7f3b08133800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:38.947+0000 7f3b08133800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:38.966 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:10:38Z"} 2024-03-04T16:10:38.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:10:42.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-03-04T16:10:43.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:43.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:43.968 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:44.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:44.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:44.069 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:44.069+0000 7f559f73b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:44.070 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:44.070+0000 7f559f73b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:44.089 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:10:44Z"} 2024-03-04T16:10:44.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:10:49.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:49.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:49.091 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:49.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:49.147 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:49.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:49.193+0000 7fd65b613800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:49.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:49.193+0000 7fd65b613800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:49.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:10:49Z"} 2024-03-04T16:10:49.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:10:54.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:54.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:54.215 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:54.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:54.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:54.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:54.314+0000 7f332e52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:54.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:54.315+0000 7f332e52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:54.334 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:10:54Z"} 2024-03-04T16:10:54.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:10:59.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:59.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:10:59.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:10:59.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:10:59.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:10:59.439 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:59.438+0000 7fbe8748c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:59.439 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:10:59.439+0000 7fbe8748c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:10:59.458 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:10:59Z"} 2024-03-04T16:10:59.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:04.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:04.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:04.460 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:04.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:04.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:04.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:04.561+0000 7fc10972f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:04.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:04.562+0000 7fc10972f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:04.581 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:04Z"} 2024-03-04T16:11:04.582 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:09.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:09.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:09.583 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:09.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:09.687+0000 7f10327a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:09.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:09.688+0000 7f10327a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:09.707 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:09Z"} 2024-03-04T16:11:09.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:14.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:14.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:14.708 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:14.762 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:14.762 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:14.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:14.808+0000 7f05e2211800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:14.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:14.809+0000 7f05e2211800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:14.829 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:14Z"} 2024-03-04T16:11:14.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:19.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:19.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:19.831 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:19.884 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:19.884 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:19.930+0000 7f603fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:19.931+0000 7f603fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:19.950 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:19Z"} 2024-03-04T16:11:19.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:24.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:24.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:24.952 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:25.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:25.054+0000 7fd7e5612800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:25.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:25.054+0000 7fd7e5612800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:25.074 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:25Z"} 2024-03-04T16:11:25.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:30.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:30.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:30.076 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:30.129 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:30.129 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:30.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:30.176+0000 7fec69a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:30.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:30.177+0000 7fec69a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:30.196 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:30Z"} 2024-03-04T16:11:30.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:35.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:35.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:35.198 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:35.299+0000 7faf1fd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:35.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:35.300+0000 7faf1fd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:35.319 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:35Z"} 2024-03-04T16:11:35.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:40.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:40.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:40.321 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:40.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:40.421+0000 7fc70ae85800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:40.424 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:40.422+0000 7fc70ae85800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:40.441 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:40Z"} 2024-03-04T16:11:40.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:45.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:45.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:45.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:45.496 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:45.497 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:45.544+0000 7fc7caa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:45.546 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:45.545+0000 7fc7caa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:45.565 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:45Z"} 2024-03-04T16:11:45.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:50.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:50.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:50.567 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:50.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:50.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:50.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:50.667+0000 7f817d614800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:50.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:50.667+0000 7f817d614800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:50.687 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:50Z"} 2024-03-04T16:11:50.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:11:55.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:55.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:11:55.688 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:11:55.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:11:55.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:11:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:55.788+0000 7f3ad188c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:11:55.789+0000 7f3ad188c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:11:55.808 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:11:55Z"} 2024-03-04T16:11:55.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:00.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:00.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:00.810 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:00.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:00.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:00.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:00.910+0000 7f8945a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:00.911+0000 7f8945a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:00.931 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:00Z"} 2024-03-04T16:12:00.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:05.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:05.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:05.933 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:05.985 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:05.985 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:06.030+0000 7f7aca015800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:06.031+0000 7f7aca015800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:06.050 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:06Z"} 2024-03-04T16:12:06.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:11.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:11.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:11.052 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:11.105 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:11.105 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:11.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:11.151+0000 7f80f0127800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:11.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:11.152+0000 7f80f0127800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:11.172 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:11Z"} 2024-03-04T16:12:11.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:16.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:16.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:16.173 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:16.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:16.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:16.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:16.275+0000 7efc52528800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:16.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:16.276+0000 7efc52528800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:16.296 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:16Z"} 2024-03-04T16:12:16.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:21.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:21.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:21.298 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:21.351 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:21.351 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:21.399+0000 7fb8e0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:21.400+0000 7fb8e0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:21.420 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:21Z"} 2024-03-04T16:12:21.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:26.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:26.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:26.421 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:26.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:26.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:26.521 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:26.522+0000 7f4ca2e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:26.522 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:26.523+0000 7f4ca2e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:26.542 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:26Z"} 2024-03-04T16:12:26.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:31.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:31.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:31.544 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:31.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:31.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:31.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:31.646+0000 7f5e39311800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:31.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:31.647+0000 7f5e39311800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:31.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:31Z"} 2024-03-04T16:12:31.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709568638.712461_34.1 target= 2024-03-04T16:12:36.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:36.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:36.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:36.768 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:36.767+0000 7f5cb6a34800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:36.769 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:36.768+0000 7f5cb6a34800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:36.788 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:36Z"} 2024-03-04T16:12:36.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:36.788 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:36.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.789 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:36.898+0000 7f804aa36800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:36.899+0000 7f804aa36800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:36.920 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:36Z"} 2024-03-04T16:12:36.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:36.921 INFO:rgw_multi.tests:meta checkpoint 2024-03-04T16:12:36.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:36.922 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:36.969 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:36.969 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:37.034 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:37.035 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:37.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:37.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:37.035 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:37.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:37.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:37.128 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:37.128+0000 7f43de20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:37.129 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:37.128+0000 7f43de20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:37.147 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:37Z"} 2024-03-04T16:12:37.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:37.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:37.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:37.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:37.148 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:37.196 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:37.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:37.242+0000 7f8ecf911800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:37.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:37.243+0000 7f8ecf911800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:37.263 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:37Z"} 2024-03-04T16:12:37.264 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:37.264 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-03-04T16:12:37.648 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.648 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.648 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.648 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.649 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.649 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.649 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.650 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.650 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.650 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.650 DEBUG:boto:Using access key provided by client. 2024-03-04T16:12:37.651 DEBUG:boto:Using secret key provided by client. 2024-03-04T16:12:37.651 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-1 2024-03-04T16:12:37.651 DEBUG:boto:path=/sbnvju-1/ 2024-03-04T16:12:37.651 DEBUG:boto:auth_path=/sbnvju-1/ 2024-03-04T16:12:37.651 DEBUG:boto:Method: PUT 2024-03-04T16:12:37.651 DEBUG:boto:Path: /sbnvju-1/ 2024-03-04T16:12:37.651 DEBUG:boto:Data: 2024-03-04T16:12:37.651 DEBUG:boto:Headers: {} 2024-03-04T16:12:37.651 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:37.652 DEBUG:boto:Port: 8000 2024-03-04T16:12:37.652 DEBUG:boto:Params: {} 2024-03-04T16:12:37.652 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:12:37.652 DEBUG:boto:Token: None 2024-03-04T16:12:37.652 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:37 GMT /sbnvju-1/ 2024-03-04T16:12:37.652 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SrfWTg4BtvOtK8eWSvUPUHFPR8s= 2024-03-04T16:12:37.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SrfWTg4BtvOtK8eWSvUPUHFPR8s=', 'Content-Length': '0'} 2024-03-04T16:12:40.459 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd24ffd6dba06c79-0065e5f2f5-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:40.459 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-2 2024-03-04T16:12:40.459 DEBUG:boto:path=/sbnvju-2/ 2024-03-04T16:12:40.459 DEBUG:boto:auth_path=/sbnvju-2/ 2024-03-04T16:12:40.459 DEBUG:boto:Method: PUT 2024-03-04T16:12:40.459 DEBUG:boto:Path: /sbnvju-2/ 2024-03-04T16:12:40.460 DEBUG:boto:Data: 2024-03-04T16:12:40.460 DEBUG:boto:Headers: {} 2024-03-04T16:12:40.460 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:40.460 DEBUG:boto:Port: 8000 2024-03-04T16:12:40.460 DEBUG:boto:Params: {} 2024-03-04T16:12:40.460 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:12:40.460 DEBUG:boto:Token: None 2024-03-04T16:12:40.460 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:40 GMT /sbnvju-2/ 2024-03-04T16:12:40.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sOSzT+5o/JBWhIH+cvhcvJxKUnw= 2024-03-04T16:12:40.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sOSzT+5o/JBWhIH+cvhcvJxKUnw=', 'Content-Length': '0'} 2024-03-04T16:12:42.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5c433f8f152389c-0065e5f2f8-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:42.964 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-3 2024-03-04T16:12:42.965 DEBUG:boto:path=/sbnvju-3/ 2024-03-04T16:12:42.965 DEBUG:boto:auth_path=/sbnvju-3/ 2024-03-04T16:12:42.965 DEBUG:boto:Method: PUT 2024-03-04T16:12:42.965 DEBUG:boto:Path: /sbnvju-3/ 2024-03-04T16:12:42.965 DEBUG:boto:Data: 2024-03-04T16:12:42.965 DEBUG:boto:Headers: {} 2024-03-04T16:12:42.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:42.966 DEBUG:boto:Port: 8001 2024-03-04T16:12:42.966 DEBUG:boto:Params: {} 2024-03-04T16:12:42.966 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:12:42.966 DEBUG:boto:Token: None 2024-03-04T16:12:42.966 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:42 GMT /sbnvju-3/ 2024-03-04T16:12:42.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4Cic12Oqn4b1N2J7t4Nqe70PeBs= 2024-03-04T16:12:42.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4Cic12Oqn4b1N2J7t4Nqe70PeBs=', 'Content-Length': '0'} 2024-03-04T16:12:43.004 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c116d9ee824682d-0065e5f2fa-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:43.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:43.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:43.005 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:43.057 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:43.057 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"1_1709568762.991336_43.1","last_update":"2024-03-04T16:12:42.991336Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:43.125 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:43.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:43.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:43.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:43.126 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:43.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:43.224+0000 7f3cd514b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:43.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:43.225+0000 7f3cd514b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:43.244 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568760.438173_32.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:40.438173Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:43Z"} 2024-03-04T16:12:43.245 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568760.438173_32.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:40.438173Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:43Z"} 2024-03-04T16:12:43.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568760.438173_32.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:40.438173Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:43.245 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:43.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568760.438173_32.1', 2: '', 3: ''} 2024-03-04T16:12:43.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1709568762.991336_43.1 target=1_1709568760.438173_32.1 2024-03-04T16:12:48.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.247 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:48.300 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:48.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:48.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:48.349+0000 7f057ba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:48.350+0000 7f057ba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.369 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568762.991336_43.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:42.991336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.370 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568762.991336_43.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:42.991336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568762.991336_43.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:42.991336Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:48.370 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:48.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:48.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:48.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:48.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.371 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:48.478+0000 7fb272a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:48.479+0000 7fb272a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.499 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568762.991336_43.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:42.991336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.499 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568762.991336_43.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:42.991336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568762.991336_43.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:42.991336Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:48.499 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:48.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568762.991336_43.1', 2: '', 3: ''} 2024-03-04T16:12:48.500 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:48.500 DEBUG:boto:path=/sbnvju-1/ 2024-03-04T16:12:48.500 DEBUG:boto:auth_path=/sbnvju-1/ 2024-03-04T16:12:48.500 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.500 DEBUG:boto:Path: /sbnvju-1/ 2024-03-04T16:12:48.500 DEBUG:boto:Data: 2024-03-04T16:12:48.500 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.500 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.501 DEBUG:boto:Params: {} 2024-03-04T16:12:48.501 DEBUG:boto:Token: None 2024-03-04T16:12:48.501 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-1/ 2024-03-04T16:12:48.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo= 2024-03-04T16:12:48.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo=', 'Content-Length': '0'} 2024-03-04T16:12:48.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f54f1ab6638cea03-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.504 DEBUG:boto:path=/sbnvju-2/ 2024-03-04T16:12:48.504 DEBUG:boto:auth_path=/sbnvju-2/ 2024-03-04T16:12:48.504 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.504 DEBUG:boto:Path: /sbnvju-2/ 2024-03-04T16:12:48.504 DEBUG:boto:Data: 2024-03-04T16:12:48.504 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.504 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.504 DEBUG:boto:Params: {} 2024-03-04T16:12:48.504 DEBUG:boto:Token: None 2024-03-04T16:12:48.505 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-2/ 2024-03-04T16:12:48.505 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw= 2024-03-04T16:12:48.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw=', 'Content-Length': '0'} 2024-03-04T16:12:48.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f09acb74c3031ce-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.507 DEBUG:boto:path=/sbnvju-3/ 2024-03-04T16:12:48.507 DEBUG:boto:auth_path=/sbnvju-3/ 2024-03-04T16:12:48.507 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.507 DEBUG:boto:Path: /sbnvju-3/ 2024-03-04T16:12:48.507 DEBUG:boto:Data: 2024-03-04T16:12:48.507 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.507 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.507 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.507 DEBUG:boto:Params: {} 2024-03-04T16:12:48.508 DEBUG:boto:Token: None 2024-03-04T16:12:48.508 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-3/ 2024-03-04T16:12:48.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw= 2024-03-04T16:12:48.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw=', 'Content-Length': '0'} 2024-03-04T16:12:48.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000969b4df781cc9611-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.510 DEBUG:boto:path=/sbnvju-1/ 2024-03-04T16:12:48.510 DEBUG:boto:auth_path=/sbnvju-1/ 2024-03-04T16:12:48.510 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.511 DEBUG:boto:Path: /sbnvju-1/ 2024-03-04T16:12:48.511 DEBUG:boto:Data: 2024-03-04T16:12:48.511 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.511 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.511 DEBUG:boto:Params: {} 2024-03-04T16:12:48.511 DEBUG:boto:Token: None 2024-03-04T16:12:48.511 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-1/ 2024-03-04T16:12:48.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo= 2024-03-04T16:12:48.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo=', 'Content-Length': '0'} 2024-03-04T16:12:48.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b268b65127b64e1-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.514 DEBUG:boto:path=/sbnvju-2/ 2024-03-04T16:12:48.514 DEBUG:boto:auth_path=/sbnvju-2/ 2024-03-04T16:12:48.514 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.514 DEBUG:boto:Path: /sbnvju-2/ 2024-03-04T16:12:48.514 DEBUG:boto:Data: 2024-03-04T16:12:48.514 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.514 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.514 DEBUG:boto:Params: {} 2024-03-04T16:12:48.515 DEBUG:boto:Token: None 2024-03-04T16:12:48.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-2/ 2024-03-04T16:12:48.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw= 2024-03-04T16:12:48.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw=', 'Content-Length': '0'} 2024-03-04T16:12:48.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000527acd922eb27d66-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.517 DEBUG:boto:path=/sbnvju-3/ 2024-03-04T16:12:48.517 DEBUG:boto:auth_path=/sbnvju-3/ 2024-03-04T16:12:48.517 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.517 DEBUG:boto:Path: /sbnvju-3/ 2024-03-04T16:12:48.518 DEBUG:boto:Data: 2024-03-04T16:12:48.518 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.518 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.518 DEBUG:boto:Params: {} 2024-03-04T16:12:48.518 DEBUG:boto:Token: None 2024-03-04T16:12:48.518 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-3/ 2024-03-04T16:12:48.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw= 2024-03-04T16:12:48.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw=', 'Content-Length': '0'} 2024-03-04T16:12:48.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfd1f133a120cb1a-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.520 DEBUG:boto:path=/sbnvju-1/ 2024-03-04T16:12:48.521 DEBUG:boto:auth_path=/sbnvju-1/ 2024-03-04T16:12:48.521 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.521 DEBUG:boto:Path: /sbnvju-1/ 2024-03-04T16:12:48.521 DEBUG:boto:Data: 2024-03-04T16:12:48.521 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.521 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.521 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.521 DEBUG:boto:Params: {} 2024-03-04T16:12:48.521 DEBUG:boto:Token: None 2024-03-04T16:12:48.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-1/ 2024-03-04T16:12:48.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo= 2024-03-04T16:12:48.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Z2DfnPpEzVFc0lfDhK5IBtblIBo=', 'Content-Length': '0'} 2024-03-04T16:12:48.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7a87b4d2dcec46b-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.524 DEBUG:boto:path=/sbnvju-2/ 2024-03-04T16:12:48.524 DEBUG:boto:auth_path=/sbnvju-2/ 2024-03-04T16:12:48.524 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.524 DEBUG:boto:Path: /sbnvju-2/ 2024-03-04T16:12:48.524 DEBUG:boto:Data: 2024-03-04T16:12:48.524 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.524 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.524 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.525 DEBUG:boto:Params: {} 2024-03-04T16:12:48.525 DEBUG:boto:Token: None 2024-03-04T16:12:48.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-2/ 2024-03-04T16:12:48.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw= 2024-03-04T16:12:48.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YbNzdDiy9ZsDF2AFJXJYdVd/Bbw=', 'Content-Length': '0'} 2024-03-04T16:12:48.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b698c285f3b59538-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.527 DEBUG:boto:path=/sbnvju-3/ 2024-03-04T16:12:48.527 DEBUG:boto:auth_path=/sbnvju-3/ 2024-03-04T16:12:48.527 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.527 DEBUG:boto:Path: /sbnvju-3/ 2024-03-04T16:12:48.527 DEBUG:boto:Data: 2024-03-04T16:12:48.527 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.528 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.528 DEBUG:boto:Params: {} 2024-03-04T16:12:48.528 DEBUG:boto:Token: None 2024-03-04T16:12:48.528 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-3/ 2024-03-04T16:12:48.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw= 2024-03-04T16:12:48.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7RHXZTF4FMQzyubYHC27GCkh9fw=', 'Content-Length': '0'} 2024-03-04T16:12:48.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f19c5716d7df1ec-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-03-04T16:12:48.531 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-4 2024-03-04T16:12:48.531 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:48.531 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:48.531 DEBUG:boto:Method: PUT 2024-03-04T16:12:48.531 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:48.532 DEBUG:boto:Data: 2024-03-04T16:12:48.532 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.532 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.532 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.532 DEBUG:boto:Params: {} 2024-03-04T16:12:48.532 DEBUG:boto:Token: None 2024-03-04T16:12:48.532 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-4/ 2024-03-04T16:12:48.532 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dcS2eLLYKSlOP53Cflj/JloFfA0= 2024-03-04T16:12:48.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dcS2eLLYKSlOP53Cflj/JloFfA0=', 'Content-Length': '0'} 2024-03-04T16:12:48.546 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062e5e3d2ba0a4008-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.546 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-5 2024-03-04T16:12:48.546 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:48.547 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:48.547 DEBUG:boto:Method: PUT 2024-03-04T16:12:48.547 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:48.547 DEBUG:boto:Data: 2024-03-04T16:12:48.547 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.547 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.547 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.547 DEBUG:boto:Params: {} 2024-03-04T16:12:48.547 DEBUG:boto:Token: None 2024-03-04T16:12:48.547 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-5/ 2024-03-04T16:12:48.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:82MlrttPGimakZyxrpAM3lIii1c= 2024-03-04T16:12:48.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:82MlrttPGimakZyxrpAM3lIii1c=', 'Content-Length': '0'} 2024-03-04T16:12:48.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083a9541f3500808d-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.574 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-6 2024-03-04T16:12:48.574 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:48.574 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:48.574 DEBUG:boto:Method: PUT 2024-03-04T16:12:48.574 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:48.574 DEBUG:boto:Data: 2024-03-04T16:12:48.575 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.575 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.575 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.575 DEBUG:boto:Params: {} 2024-03-04T16:12:48.575 DEBUG:boto:Token: None 2024-03-04T16:12:48.575 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-6/ 2024-03-04T16:12:48.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SMReEnKLxb53+Kkku70IqolEOaw= 2024-03-04T16:12:48.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SMReEnKLxb53+Kkku70IqolEOaw=', 'Content-Length': '0'} 2024-03-04T16:12:48.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000126c21dba895707e-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.598 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:48.644 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:48.645 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:48.731 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"1_1709568768.589812_55.1","last_update":"2024-03-04T16:12:48.589812Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:48.731 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568638.712461_34.1', 1: '1_1709568768.589812_55.1', 2: '', 3: ''} 2024-03-04T16:12:48.732 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:48.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.732 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:48.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:48.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:48.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:48.826+0000 7f3670288800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:48.827+0000 7f3670288800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568768.589812_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:48.589812Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.847 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568768.589812_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:48.589812Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568768.589812_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:48.589812Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:48.847 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568768.589812_55.1', 2: '', 3: ''} 2024-03-04T16:12:48.847 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568768.589812_55.1', 2: '', 3: ''} 2024-03-04T16:12:48.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:48.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:48.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:48.848 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:48.940+0000 7f4e0d9a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:48.941+0000 7f4e0d9a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:48.966 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568768.589812_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:48.589812Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.967 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568768.589812_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:48.589812Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:48Z"} 2024-03-04T16:12:48.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568768.589812_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:48.589812Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:48.967 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568768.589812_55.1', 2: '', 3: ''} 2024-03-04T16:12:48.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568768.589812_55.1', 2: '', 3: ''} 2024-03-04T16:12:48.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:48.967 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:48.967 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:48.968 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.968 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:48.968 DEBUG:boto:Data: 2024-03-04T16:12:48.968 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.968 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.968 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.968 DEBUG:boto:Params: {} 2024-03-04T16:12:48.968 DEBUG:boto:Token: None 2024-03-04T16:12:48.968 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-4/ 2024-03-04T16:12:48.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8= 2024-03-04T16:12:48.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8=', 'Content-Length': '0'} 2024-03-04T16:12:48.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044ae4a828a839e44-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.971 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:48.971 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:48.972 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.972 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:48.972 DEBUG:boto:Data: 2024-03-04T16:12:48.972 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.972 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.972 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.972 DEBUG:boto:Params: {} 2024-03-04T16:12:48.972 DEBUG:boto:Token: None 2024-03-04T16:12:48.973 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-5/ 2024-03-04T16:12:48.973 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I= 2024-03-04T16:12:48.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I=', 'Content-Length': '0'} 2024-03-04T16:12:48.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b528c1b038e81588-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.975 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:48.975 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:48.976 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.976 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:48.976 DEBUG:boto:Data: 2024-03-04T16:12:48.976 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.976 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:48.976 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.976 DEBUG:boto:Params: {} 2024-03-04T16:12:48.977 DEBUG:boto:Token: None 2024-03-04T16:12:48.977 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-6/ 2024-03-04T16:12:48.977 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA= 2024-03-04T16:12:48.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA=', 'Content-Length': '0'} 2024-03-04T16:12:48.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b89c48cf5685c3f-0065e5f300-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.979 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:48.979 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:48.979 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.979 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:48.979 DEBUG:boto:Data: 2024-03-04T16:12:48.979 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.979 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.979 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.979 DEBUG:boto:Params: {} 2024-03-04T16:12:48.980 DEBUG:boto:Token: None 2024-03-04T16:12:48.980 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-4/ 2024-03-04T16:12:48.980 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8= 2024-03-04T16:12:48.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8=', 'Content-Length': '0'} 2024-03-04T16:12:48.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000007aeadf09b8439a-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.982 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:48.982 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:48.982 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.983 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:48.983 DEBUG:boto:Data: 2024-03-04T16:12:48.983 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.983 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.983 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.983 DEBUG:boto:Params: {} 2024-03-04T16:12:48.983 DEBUG:boto:Token: None 2024-03-04T16:12:48.984 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-5/ 2024-03-04T16:12:48.984 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I= 2024-03-04T16:12:48.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I=', 'Content-Length': '0'} 2024-03-04T16:12:48.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000701da88095ce31e3-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.986 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:48.986 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:48.986 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.986 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:48.987 DEBUG:boto:Data: 2024-03-04T16:12:48.987 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.987 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:48.987 DEBUG:boto:Port: 8000 2024-03-04T16:12:48.987 DEBUG:boto:Params: {} 2024-03-04T16:12:48.987 DEBUG:boto:Token: None 2024-03-04T16:12:48.987 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-6/ 2024-03-04T16:12:48.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA= 2024-03-04T16:12:48.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA=', 'Content-Length': '0'} 2024-03-04T16:12:48.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8e59ba49107899b-0065e5f300-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.989 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:48.989 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:48.989 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.990 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:48.990 DEBUG:boto:Data: 2024-03-04T16:12:48.990 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.990 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.990 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.990 DEBUG:boto:Params: {} 2024-03-04T16:12:48.990 DEBUG:boto:Token: None 2024-03-04T16:12:48.990 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-4/ 2024-03-04T16:12:48.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8= 2024-03-04T16:12:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uFetue/FiUeZehy7fo+k+hMevU8=', 'Content-Length': '0'} 2024-03-04T16:12:48.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000854f4558196867b8-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.993 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:48.993 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:48.993 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.993 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:48.993 DEBUG:boto:Data: 2024-03-04T16:12:48.993 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.993 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.994 DEBUG:boto:Params: {} 2024-03-04T16:12:48.994 DEBUG:boto:Token: None 2024-03-04T16:12:48.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-5/ 2024-03-04T16:12:48.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I= 2024-03-04T16:12:48.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ggyzr7+WHam0Sy/6aEpAVzIl/8I=', 'Content-Length': '0'} 2024-03-04T16:12:48.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c19f6cad8cb589bf-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:48.997 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:48.997 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:48.997 DEBUG:boto:Method: HEAD 2024-03-04T16:12:48.997 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:48.997 DEBUG:boto:Data: 2024-03-04T16:12:48.997 DEBUG:boto:Headers: {} 2024-03-04T16:12:48.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:48.997 DEBUG:boto:Port: 8001 2024-03-04T16:12:48.997 DEBUG:boto:Params: {} 2024-03-04T16:12:48.998 DEBUG:boto:Token: None 2024-03-04T16:12:48.998 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:48 GMT /sbnvju-6/ 2024-03-04T16:12:48.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA= 2024-03-04T16:12:48.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gg5LUD047NG/ri3jQIUfwpOZsjA=', 'Content-Length': '0'} 2024-03-04T16:12:49.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047d156d9e1f5f947-0065e5f300-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.000 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.000 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.000 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.000 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.000 DEBUG:boto:Data: 2024-03-04T16:12:49.001 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.001 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.001 DEBUG:boto:Params: {} 2024-03-04T16:12:49.001 DEBUG:boto:Token: None 2024-03-04T16:12:49.001 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk= 2024-03-04T16:12:49.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk=', 'Content-Length': '0'} 2024-03-04T16:12:49.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065a54d1f78411a1c-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.020 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.020 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.020 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.020 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.020 DEBUG:boto:Data: 2024-03-04T16:12:49.020 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.022 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.022 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.023 DEBUG:boto:Params: {} 2024-03-04T16:12:49.023 DEBUG:boto:Token: None 2024-03-04T16:12:49.023 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.023 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c= 2024-03-04T16:12:49.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c=', 'Content-Length': '0'} 2024-03-04T16:12:49.040 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a8a66809e77182b-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.040 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.040 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.040 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.041 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.041 DEBUG:boto:Data: 2024-03-04T16:12:49.041 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.041 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.041 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.041 DEBUG:boto:Params: {} 2024-03-04T16:12:49.041 DEBUG:boto:Token: None 2024-03-04T16:12:49.041 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.042 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48= 2024-03-04T16:12:49.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48=', 'Content-Length': '0'} 2024-03-04T16:12:49.066 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe028d673b1cad1a-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.067 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.067 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.067 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.067 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.067 DEBUG:boto:Data: 2024-03-04T16:12:49.067 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.067 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.067 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.067 DEBUG:boto:Params: {} 2024-03-04T16:12:49.068 DEBUG:boto:Token: None 2024-03-04T16:12:49.068 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.068 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk= 2024-03-04T16:12:49.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk=', 'Content-Length': '0'} 2024-03-04T16:12:49.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b0246f978fa8df9-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.092 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.092 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.092 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.092 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.092 DEBUG:boto:Data: 2024-03-04T16:12:49.092 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.092 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.092 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.093 DEBUG:boto:Params: {} 2024-03-04T16:12:49.093 DEBUG:boto:Token: None 2024-03-04T16:12:49.093 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c= 2024-03-04T16:12:49.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c=', 'Content-Length': '0'} 2024-03-04T16:12:49.165 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db60a0e8dc492fdd-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.165 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.166 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.166 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.166 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.166 DEBUG:boto:Data: 2024-03-04T16:12:49.166 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.166 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.166 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.167 DEBUG:boto:Params: {} 2024-03-04T16:12:49.167 DEBUG:boto:Token: None 2024-03-04T16:12:49.167 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.167 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48= 2024-03-04T16:12:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48=', 'Content-Length': '0'} 2024-03-04T16:12:49.245 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c27ca1427e8efcb4-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.245 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.245 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.245 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.245 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.245 DEBUG:boto:Data: 2024-03-04T16:12:49.245 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.246 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.246 DEBUG:boto:Params: {} 2024-03-04T16:12:49.246 DEBUG:boto:Token: None 2024-03-04T16:12:49.246 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk= 2024-03-04T16:12:49.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Ljyk8FymhpKLRqZ7rRls3BoQjk=', 'Content-Length': '0'} 2024-03-04T16:12:49.282 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1324288b0967c06-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.282 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.282 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.283 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.283 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.283 DEBUG:boto:Data: 2024-03-04T16:12:49.283 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.283 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.283 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.283 DEBUG:boto:Params: {} 2024-03-04T16:12:49.283 DEBUG:boto:Token: None 2024-03-04T16:12:49.283 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c= 2024-03-04T16:12:49.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EVtJQxj4oGiCVaxFkmXwU76FL2c=', 'Content-Length': '0'} 2024-03-04T16:12:49.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4e5f95fd919b107-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.348 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.349 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.349 DEBUG:boto:Method: PUT 2024-03-04T16:12:49.349 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.349 DEBUG:boto:Data: 2024-03-04T16:12:49.349 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.349 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.349 DEBUG:boto:Params: {} 2024-03-04T16:12:49.349 DEBUG:boto:Token: None 2024-03-04T16:12:49.349 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48= 2024-03-04T16:12:49.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kxtA+iEf/cGvPYE9mswqQKJrx48=', 'Content-Length': '0'} 2024-03-04T16:12:49.418 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044f34eb8ef9eb9cc-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.419 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.419 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.419 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.419 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.419 DEBUG:boto:Data: 2024-03-04T16:12:49.419 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.419 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.419 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.419 DEBUG:boto:Params: {} 2024-03-04T16:12:49.419 DEBUG:boto:Token: None 2024-03-04T16:12:49.420 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.420 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc= 2024-03-04T16:12:49.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc=', 'Content-Length': '0'} 2024-03-04T16:12:49.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000966b37f2d8a1eb13-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.422 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.423 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.423 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.423 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.423 DEBUG:boto:Data: 2024-03-04T16:12:49.423 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.423 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.423 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.423 DEBUG:boto:Params: {} 2024-03-04T16:12:49.423 DEBUG:boto:Token: None 2024-03-04T16:12:49.423 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.424 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g= 2024-03-04T16:12:49.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g=', 'Content-Length': '0'} 2024-03-04T16:12:49.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c09a51aca485247-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.426 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.426 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.426 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.426 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.426 DEBUG:boto:Data: 2024-03-04T16:12:49.426 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.427 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:49.427 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.427 DEBUG:boto:Params: {} 2024-03-04T16:12:49.427 DEBUG:boto:Token: None 2024-03-04T16:12:49.427 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.427 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M= 2024-03-04T16:12:49.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M=', 'Content-Length': '0'} 2024-03-04T16:12:49.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026a8f7b3a3991f56-0065e5f301-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.429 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.430 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.430 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.430 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.430 DEBUG:boto:Data: 2024-03-04T16:12:49.430 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.430 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.430 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.430 DEBUG:boto:Params: {} 2024-03-04T16:12:49.430 DEBUG:boto:Token: None 2024-03-04T16:12:49.430 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.431 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc= 2024-03-04T16:12:49.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc=', 'Content-Length': '0'} 2024-03-04T16:12:49.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b108451367bfb4ea-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.433 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.433 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.433 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.433 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.433 DEBUG:boto:Data: 2024-03-04T16:12:49.433 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.434 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.434 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.434 DEBUG:boto:Params: {} 2024-03-04T16:12:49.434 DEBUG:boto:Token: None 2024-03-04T16:12:49.434 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.434 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g= 2024-03-04T16:12:49.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g=', 'Content-Length': '0'} 2024-03-04T16:12:49.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be858d5569f06bd4-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.436 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.437 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.437 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.437 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.437 DEBUG:boto:Data: 2024-03-04T16:12:49.437 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:49.437 DEBUG:boto:Port: 8000 2024-03-04T16:12:49.437 DEBUG:boto:Params: {} 2024-03-04T16:12:49.437 DEBUG:boto:Token: None 2024-03-04T16:12:49.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.438 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M= 2024-03-04T16:12:49.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M=', 'Content-Length': '0'} 2024-03-04T16:12:49.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4ef4a1a001a246b-0065e5f301-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.440 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:49.440 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:49.440 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.440 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:49.441 DEBUG:boto:Data: 2024-03-04T16:12:49.441 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.441 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.441 DEBUG:boto:Params: {} 2024-03-04T16:12:49.441 DEBUG:boto:Token: None 2024-03-04T16:12:49.441 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-4/ 2024-03-04T16:12:49.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc= 2024-03-04T16:12:49.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TU282cdiw4q+lLYW3AYByVlgOvc=', 'Content-Length': '0'} 2024-03-04T16:12:49.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000198fb76cee682860-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.444 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:49.444 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:49.444 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.444 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:49.444 DEBUG:boto:Data: 2024-03-04T16:12:49.444 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.444 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.444 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.444 DEBUG:boto:Params: {} 2024-03-04T16:12:49.444 DEBUG:boto:Token: None 2024-03-04T16:12:49.445 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-5/ 2024-03-04T16:12:49.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g= 2024-03-04T16:12:49.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UNFc6LwCZMQ00+uLJMwOrLJaZ3g=', 'Content-Length': '0'} 2024-03-04T16:12:49.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000731adaf183ce9c9e-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.447 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:49.447 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:49.447 DEBUG:boto:Method: HEAD 2024-03-04T16:12:49.447 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:49.447 DEBUG:boto:Data: 2024-03-04T16:12:49.448 DEBUG:boto:Headers: {} 2024-03-04T16:12:49.448 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:49.448 DEBUG:boto:Port: 8001 2024-03-04T16:12:49.448 DEBUG:boto:Params: {} 2024-03-04T16:12:49.448 DEBUG:boto:Token: None 2024-03-04T16:12:49.448 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:49 GMT /sbnvju-6/ 2024-03-04T16:12:49.448 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M= 2024-03-04T16:12:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9D+3rkH9mHFVPL2/7lDk5+v7N3M=', 'Content-Length': '0'} 2024-03-04T16:12:49.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000675e3961189aeb4f-0065e5f301-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:49.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:49.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:49.452 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:49.499 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:49.499 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:49.563 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"1_1709568769.363344_91.1","last_update":"2024-03-04T16:12:49.363344Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:49.564 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:49.564 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:49.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:49.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:49.564 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:49.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:49.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:49.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:49.656+0000 7f2d9551f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:49.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:49.657+0000 7f2d9551f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:49.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.075122_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.075122Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:49Z"} 2024-03-04T16:12:49.676 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.075122_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.075122Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:49Z"} 2024-03-04T16:12:49.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568769.075122_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:49.075122Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:49.676 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:49.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.075122_68.1', 2: '', 3: ''} 2024-03-04T16:12:49.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1709568769.363344_91.1 target=1_1709568769.075122_68.1 2024-03-04T16:12:54.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:54.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:54.678 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:54.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:54.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:54.779+0000 7fba8723a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:54.780+0000 7fba8723a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:54.800 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.363344_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.363344Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:54Z"} 2024-03-04T16:12:54.800 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.363344_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.363344Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:54Z"} 2024-03-04T16:12:54.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568769.363344_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:49.363344Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:54.801 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:54.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:54.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:54.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:54.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:54.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:54.802 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:54.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:54.905+0000 7f76e4d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:54.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:54.907+0000 7f76e4d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:54.927 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.363344_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.363344Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:54Z"} 2024-03-04T16:12:54.927 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568769.363344_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:49.363344Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:54Z"} 2024-03-04T16:12:54.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568769.363344_91.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:49.363344Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:54.927 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:54.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568769.363344_91.1', 2: '', 3: ''} 2024-03-04T16:12:54.928 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:54.928 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:54.928 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:54.928 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.928 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:54.928 DEBUG:boto:Data: 2024-03-04T16:12:54.928 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.928 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:54.928 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.928 DEBUG:boto:Params: {} 2024-03-04T16:12:54.929 DEBUG:boto:Token: None 2024-03-04T16:12:54.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-4/ 2024-03-04T16:12:54.929 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64= 2024-03-04T16:12:54.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64=', 'Content-Length': '0'} 2024-03-04T16:12:54.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d4480882393710a-0065e5f306-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.932 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:54.932 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:54.932 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.932 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:54.932 DEBUG:boto:Data: 2024-03-04T16:12:54.932 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.932 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:54.932 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.932 DEBUG:boto:Params: {} 2024-03-04T16:12:54.932 DEBUG:boto:Token: None 2024-03-04T16:12:54.933 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-5/ 2024-03-04T16:12:54.933 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E= 2024-03-04T16:12:54.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E=', 'Content-Length': '0'} 2024-03-04T16:12:54.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003473a61cd76f947b-0065e5f306-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.935 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:54.935 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:54.935 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.935 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:54.935 DEBUG:boto:Data: 2024-03-04T16:12:54.935 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.935 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:54.935 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.936 DEBUG:boto:Params: {} 2024-03-04T16:12:54.936 DEBUG:boto:Token: None 2024-03-04T16:12:54.936 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-6/ 2024-03-04T16:12:54.936 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE= 2024-03-04T16:12:54.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE=', 'Content-Length': '0'} 2024-03-04T16:12:54.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008008861c5f1966f6-0065e5f306-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.938 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:54.938 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:54.938 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.938 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:54.938 DEBUG:boto:Data: 2024-03-04T16:12:54.938 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.939 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:54.939 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.939 DEBUG:boto:Params: {} 2024-03-04T16:12:54.939 DEBUG:boto:Token: None 2024-03-04T16:12:54.939 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-4/ 2024-03-04T16:12:54.939 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64= 2024-03-04T16:12:54.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64=', 'Content-Length': '0'} 2024-03-04T16:12:54.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3851b7ae18d5c6b-0065e5f306-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.944 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:54.944 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:54.944 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.944 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:54.944 DEBUG:boto:Data: 2024-03-04T16:12:54.944 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.944 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:54.944 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.944 DEBUG:boto:Params: {} 2024-03-04T16:12:54.944 DEBUG:boto:Token: None 2024-03-04T16:12:54.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-5/ 2024-03-04T16:12:54.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E= 2024-03-04T16:12:54.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E=', 'Content-Length': '0'} 2024-03-04T16:12:54.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b26e22316f57fb05-0065e5f306-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.947 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:54.947 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:54.947 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.948 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:54.948 DEBUG:boto:Data: 2024-03-04T16:12:54.948 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.948 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:54.948 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.948 DEBUG:boto:Params: {} 2024-03-04T16:12:54.948 DEBUG:boto:Token: None 2024-03-04T16:12:54.948 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-6/ 2024-03-04T16:12:54.948 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE= 2024-03-04T16:12:54.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE=', 'Content-Length': '0'} 2024-03-04T16:12:54.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e581b06db4dcc64-0065e5f306-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.950 DEBUG:boto:path=/sbnvju-4/ 2024-03-04T16:12:54.951 DEBUG:boto:auth_path=/sbnvju-4/ 2024-03-04T16:12:54.951 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.951 DEBUG:boto:Path: /sbnvju-4/ 2024-03-04T16:12:54.951 DEBUG:boto:Data: 2024-03-04T16:12:54.951 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.951 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:54.951 DEBUG:boto:Port: 8001 2024-03-04T16:12:54.951 DEBUG:boto:Params: {} 2024-03-04T16:12:54.951 DEBUG:boto:Token: None 2024-03-04T16:12:54.952 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-4/ 2024-03-04T16:12:54.952 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64= 2024-03-04T16:12:54.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EfEqXXz55x++O06ji+wUMgsyn64=', 'Content-Length': '0'} 2024-03-04T16:12:54.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4b3936c205ba7c2-0065e5f306-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.954 DEBUG:boto:path=/sbnvju-5/ 2024-03-04T16:12:54.954 DEBUG:boto:auth_path=/sbnvju-5/ 2024-03-04T16:12:54.955 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.955 DEBUG:boto:Path: /sbnvju-5/ 2024-03-04T16:12:54.955 DEBUG:boto:Data: 2024-03-04T16:12:54.955 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.955 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:54.955 DEBUG:boto:Port: 8001 2024-03-04T16:12:54.955 DEBUG:boto:Params: {} 2024-03-04T16:12:54.955 DEBUG:boto:Token: None 2024-03-04T16:12:54.955 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-5/ 2024-03-04T16:12:54.955 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E= 2024-03-04T16:12:54.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J89X2CGgW3AUUw1/jhmMKXQ4n/E=', 'Content-Length': '0'} 2024-03-04T16:12:54.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091dc47206214920d-0065e5f306-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.958 DEBUG:boto:path=/sbnvju-6/ 2024-03-04T16:12:54.958 DEBUG:boto:auth_path=/sbnvju-6/ 2024-03-04T16:12:54.958 DEBUG:boto:Method: HEAD 2024-03-04T16:12:54.958 DEBUG:boto:Path: /sbnvju-6/ 2024-03-04T16:12:54.958 DEBUG:boto:Data: 2024-03-04T16:12:54.958 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.958 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:54.958 DEBUG:boto:Port: 8001 2024-03-04T16:12:54.958 DEBUG:boto:Params: {} 2024-03-04T16:12:54.959 DEBUG:boto:Token: None 2024-03-04T16:12:54.959 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-6/ 2024-03-04T16:12:54.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE= 2024-03-04T16:12:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DJOOFAy41Wo2TXuDM51jdHK5nxE=', 'Content-Length': '0'} 2024-03-04T16:12:54.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7501adddaeadbed-0065e5f306-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-03-04T16:12:54.962 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-7 2024-03-04T16:12:54.962 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:12:54.962 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:12:54.962 DEBUG:boto:Method: PUT 2024-03-04T16:12:54.962 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:12:54.962 DEBUG:boto:Data: 2024-03-04T16:12:54.962 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.963 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:54.963 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.963 DEBUG:boto:Params: {} 2024-03-04T16:12:54.963 DEBUG:boto:Token: None 2024-03-04T16:12:54.963 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-7/ 2024-03-04T16:12:54.963 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aHARKEij2VE9QXI5MRXYreh90AY= 2024-03-04T16:12:54.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHARKEij2VE9QXI5MRXYreh90AY=', 'Content-Length': '0'} 2024-03-04T16:12:54.978 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025e2cb458193fcbf-0065e5f306-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:54.979 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-8 2024-03-04T16:12:54.979 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:12:54.979 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:12:54.979 DEBUG:boto:Method: PUT 2024-03-04T16:12:54.979 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:12:54.979 DEBUG:boto:Data: 2024-03-04T16:12:54.979 DEBUG:boto:Headers: {} 2024-03-04T16:12:54.979 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:54.979 DEBUG:boto:Port: 8000 2024-03-04T16:12:54.980 DEBUG:boto:Params: {} 2024-03-04T16:12:54.980 DEBUG:boto:Token: None 2024-03-04T16:12:54.980 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:54 GMT /sbnvju-8/ 2024-03-04T16:12:54.980 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4AfGyaPHpew3YtaYD1d7/V/zmlc= 2024-03-04T16:12:54.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4AfGyaPHpew3YtaYD1d7/V/zmlc=', 'Content-Length': '0'} 2024-03-04T16:12:55.009 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd8cd28b5a761509-0065e5f306-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.009 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-9 2024-03-04T16:12:55.009 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:12:55.009 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:12:55.009 DEBUG:boto:Method: PUT 2024-03-04T16:12:55.010 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:12:55.010 DEBUG:boto:Data: 2024-03-04T16:12:55.010 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.010 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:55.010 DEBUG:boto:Port: 8001 2024-03-04T16:12:55.010 DEBUG:boto:Params: {} 2024-03-04T16:12:55.010 DEBUG:boto:Token: None 2024-03-04T16:12:55.010 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-9/ 2024-03-04T16:12:55.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Afkj6lb1QcQgIEqOQUBT66UuGAk= 2024-03-04T16:12:55.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Afkj6lb1QcQgIEqOQUBT66UuGAk=', 'Content-Length': '0'} 2024-03-04T16:12:55.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099aef3a47342043b-0065e5f307-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.033 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:55.156 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"1_1709568775.024360_104.1","last_update":"2024-03-04T16:12:55.024360Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:55.156 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568638.712461_34.1', 1: '1_1709568775.024360_104.1', 2: '', 3: ''} 2024-03-04T16:12:55.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:55.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.157 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:55.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:55.246+0000 7f0744813800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.247 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:55.246+0000 7f0744813800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.265 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.024360_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.024360Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.266 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.024360_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.024360Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.024360_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.024360Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:55.266 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.024360_104.1', 2: '', 3: ''} 2024-03-04T16:12:55.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.024360_104.1', 2: '', 3: ''} 2024-03-04T16:12:55.267 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:12:55.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:12:55.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.268 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:55.363+0000 7fcbba016800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:12:55.364+0000 7fcbba016800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.383 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.024360_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.024360Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.383 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.024360_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.024360Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.024360_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.024360Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:55.384 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.024360_104.1', 2: '', 3: ''} 2024-03-04T16:12:55.384 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.024360_104.1', 2: '', 3: ''} 2024-03-04T16:12:55.384 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:12:55.384 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:12:55.384 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:12:55.384 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.384 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:12:55.384 DEBUG:boto:Data: 2024-03-04T16:12:55.384 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:55.385 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.385 DEBUG:boto:Params: {} 2024-03-04T16:12:55.385 DEBUG:boto:Token: None 2024-03-04T16:12:55.385 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-7/ 2024-03-04T16:12:55.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc= 2024-03-04T16:12:55.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc=', 'Content-Length': '0'} 2024-03-04T16:12:55.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f54e2e77ea775cc-0065e5f307-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.388 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:12:55.388 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:12:55.388 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.388 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:12:55.388 DEBUG:boto:Data: 2024-03-04T16:12:55.388 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.388 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:55.388 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.388 DEBUG:boto:Params: {} 2024-03-04T16:12:55.388 DEBUG:boto:Token: None 2024-03-04T16:12:55.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-8/ 2024-03-04T16:12:55.389 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU= 2024-03-04T16:12:55.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU=', 'Content-Length': '0'} 2024-03-04T16:12:55.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003afd4c94ab6921e0-0065e5f307-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.391 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:12:55.391 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:12:55.392 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.392 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:12:55.392 DEBUG:boto:Data: 2024-03-04T16:12:55.392 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.392 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:55.392 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.392 DEBUG:boto:Params: {} 2024-03-04T16:12:55.392 DEBUG:boto:Token: None 2024-03-04T16:12:55.392 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-9/ 2024-03-04T16:12:55.393 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q= 2024-03-04T16:12:55.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q=', 'Content-Length': '0'} 2024-03-04T16:12:55.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa22a36567a2e78b-0065e5f307-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.394 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:12:55.395 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:12:55.395 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.395 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:12:55.395 DEBUG:boto:Data: 2024-03-04T16:12:55.395 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:55.395 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.395 DEBUG:boto:Params: {} 2024-03-04T16:12:55.395 DEBUG:boto:Token: None 2024-03-04T16:12:55.395 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-7/ 2024-03-04T16:12:55.396 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc= 2024-03-04T16:12:55.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc=', 'Content-Length': '0'} 2024-03-04T16:12:55.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b08beaf0a1b6b6f-0065e5f307-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.398 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:12:55.398 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:12:55.398 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.398 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:12:55.398 DEBUG:boto:Data: 2024-03-04T16:12:55.399 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.399 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:55.399 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.399 DEBUG:boto:Params: {} 2024-03-04T16:12:55.399 DEBUG:boto:Token: None 2024-03-04T16:12:55.399 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-8/ 2024-03-04T16:12:55.399 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU= 2024-03-04T16:12:55.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU=', 'Content-Length': '0'} 2024-03-04T16:12:55.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000117519cb780ceae1-0065e5f307-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.401 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:12:55.402 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:12:55.402 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.402 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:12:55.402 DEBUG:boto:Data: 2024-03-04T16:12:55.402 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.402 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:55.402 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.402 DEBUG:boto:Params: {} 2024-03-04T16:12:55.402 DEBUG:boto:Token: None 2024-03-04T16:12:55.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-9/ 2024-03-04T16:12:55.403 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q= 2024-03-04T16:12:55.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q=', 'Content-Length': '0'} 2024-03-04T16:12:55.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc023b14fce4a7b3-0065e5f307-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.405 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:12:55.405 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:12:55.405 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.405 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:12:55.405 DEBUG:boto:Data: 2024-03-04T16:12:55.405 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:55.406 DEBUG:boto:Port: 8001 2024-03-04T16:12:55.406 DEBUG:boto:Params: {} 2024-03-04T16:12:55.406 DEBUG:boto:Token: None 2024-03-04T16:12:55.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-7/ 2024-03-04T16:12:55.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc= 2024-03-04T16:12:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DwYApqKY09JG2ao2jWrJCdHLIpc=', 'Content-Length': '0'} 2024-03-04T16:12:55.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad7a91e45e18d8af-0065e5f307-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.408 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:12:55.408 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:12:55.408 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.408 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:12:55.409 DEBUG:boto:Data: 2024-03-04T16:12:55.409 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:55.409 DEBUG:boto:Port: 8001 2024-03-04T16:12:55.409 DEBUG:boto:Params: {} 2024-03-04T16:12:55.409 DEBUG:boto:Token: None 2024-03-04T16:12:55.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-8/ 2024-03-04T16:12:55.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU= 2024-03-04T16:12:55.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3QvGssdpljCFBx2KlLsNuIU2pNU=', 'Content-Length': '0'} 2024-03-04T16:12:55.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3f28c6381f73b36-0065e5f307-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.411 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:12:55.411 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:12:55.412 DEBUG:boto:Method: HEAD 2024-03-04T16:12:55.412 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:12:55.412 DEBUG:boto:Data: 2024-03-04T16:12:55.412 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.412 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:55.412 DEBUG:boto:Port: 8001 2024-03-04T16:12:55.412 DEBUG:boto:Params: {} 2024-03-04T16:12:55.412 DEBUG:boto:Token: None 2024-03-04T16:12:55.412 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-9/ 2024-03-04T16:12:55.412 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q= 2024-03-04T16:12:55.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QW+nrITQuMtBTelFzZYxycxxz/Q=', 'Content-Length': '0'} 2024-03-04T16:12:55.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1a704ea31cddca4-0065e5f307-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.415 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:12:55.415 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:12:55.415 DEBUG:boto:Method: DELETE 2024-03-04T16:12:55.415 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:12:55.415 DEBUG:boto:Data: 2024-03-04T16:12:55.415 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.415 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:12:55.415 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.415 DEBUG:boto:Params: {} 2024-03-04T16:12:55.416 DEBUG:boto:Token: None 2024-03-04T16:12:55.416 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-7/ 2024-03-04T16:12:55.416 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:piCffwWPhuny+pzlRHmi/gdkMKI= 2024-03-04T16:12:55.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:piCffwWPhuny+pzlRHmi/gdkMKI=', 'Content-Length': '0'} 2024-03-04T16:12:55.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ac0071dff27cc5a-0065e5f307-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.437 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:12:55.437 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:12:55.437 DEBUG:boto:Method: DELETE 2024-03-04T16:12:55.437 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:12:55.437 DEBUG:boto:Data: 2024-03-04T16:12:55.437 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:12:55.437 DEBUG:boto:Port: 8000 2024-03-04T16:12:55.438 DEBUG:boto:Params: {} 2024-03-04T16:12:55.438 DEBUG:boto:Token: None 2024-03-04T16:12:55.438 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-8/ 2024-03-04T16:12:55.438 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ax7qkr18cqnitrNhQ4a/2RE1cK8= 2024-03-04T16:12:55.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ax7qkr18cqnitrNhQ4a/2RE1cK8=', 'Content-Length': '0'} 2024-03-04T16:12:55.486 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ff2dfc284b5d9ca-0065e5f307-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.487 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:12:55.487 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:12:55.487 DEBUG:boto:Method: DELETE 2024-03-04T16:12:55.487 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:12:55.487 DEBUG:boto:Data: 2024-03-04T16:12:55.487 DEBUG:boto:Headers: {} 2024-03-04T16:12:55.487 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:12:55.487 DEBUG:boto:Port: 8001 2024-03-04T16:12:55.487 DEBUG:boto:Params: {} 2024-03-04T16:12:55.487 DEBUG:boto:Token: None 2024-03-04T16:12:55.488 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:12:55 GMT /sbnvju-9/ 2024-03-04T16:12:55.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hm89Do9qerYnkFyzUdRXfXzU+xs= 2024-03-04T16:12:55.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:12:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hm89Do9qerYnkFyzUdRXfXzU+xs=', 'Content-Length': '0'} 2024-03-04T16:12:55.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7cc039a66d3d377-0065e5f307-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:12:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:12:55.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.536 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:55.582 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:55.582 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:12:55.644 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568638.712461_34.1","last_update":"2024-03-04T16:10:38.712461Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:12:55.645 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:12:55.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:12:55.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:12:55.645 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:12:55.692 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:12:55.692 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:12:55.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:55.734+0000 7f571b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:12:55.735+0000 7f571b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:12:55.753 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.463621_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.463621Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.754 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.463621_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.463621Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:12:55Z"} 2024-03-04T16:12:55.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.463621_107.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.463621Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:12:55.754 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:12:55.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.463621_107.1', 2: '', 3: ''} 2024-03-04T16:12:55.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1709568775.513713_110.1 target=1_1709568775.463621_107.1 2024-03-04T16:13:00.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:00.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:00.756 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:13:00.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:00.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:00.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:00.858+0000 7f9d37844800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:00.859 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:00.859+0000 7f9d37844800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:00.879 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:00Z"} 2024-03-04T16:13:00.880 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:00Z"} 2024-03-04T16:13:00.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:13:00.880 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:00.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:00.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:13:00.880 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:13:00.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:00.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:00.881 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:13:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:00.986+0000 7f0751f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:00.987+0000 7f0751f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:01.008 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:00Z"} 2024-03-04T16:13:01.008 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568638.712461_34.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:10:38.712461Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:00Z"} 2024-03-04T16:13:01.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568638.712461_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:10:38.712461Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:13:01.008 DEBUG:rgw_multi.tests:log_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:01.008 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568638.712461_34.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:01.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:13:01.009 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:13:01.009 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:13:01.009 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.009 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:13:01.009 DEBUG:boto:Data: 2024-03-04T16:13:01.009 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.009 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.009 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.009 DEBUG:boto:Params: {} 2024-03-04T16:13:01.009 DEBUG:boto:Token: None 2024-03-04T16:13:01.010 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-7/ 2024-03-04T16:13:01.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4= 2024-03-04T16:13:01.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4=', 'Content-Length': '0'} 2024-03-04T16:13:01.011 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e01e8e3a6ba082c9-0065e5f30d-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.012 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:13:01.012 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:13:01.012 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.012 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:13:01.012 DEBUG:boto:Data: 2024-03-04T16:13:01.012 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.012 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.012 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.012 DEBUG:boto:Params: {} 2024-03-04T16:13:01.012 DEBUG:boto:Token: None 2024-03-04T16:13:01.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-8/ 2024-03-04T16:13:01.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY= 2024-03-04T16:13:01.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY=', 'Content-Length': '0'} 2024-03-04T16:13:01.014 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e8cbd6dd19e2f3ce-0065e5f30d-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.014 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:13:01.014 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:13:01.014 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.014 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:13:01.014 DEBUG:boto:Data: 2024-03-04T16:13:01.014 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.014 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.014 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.015 DEBUG:boto:Params: {} 2024-03-04T16:13:01.015 DEBUG:boto:Token: None 2024-03-04T16:13:01.015 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-9/ 2024-03-04T16:13:01.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c= 2024-03-04T16:13:01.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c=', 'Content-Length': '0'} 2024-03-04T16:13:01.016 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e3aae3d0c1fd476e-0065e5f30d-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.016 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:13:01.016 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:13:01.016 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.016 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:13:01.016 DEBUG:boto:Data: 2024-03-04T16:13:01.016 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.017 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.017 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.017 DEBUG:boto:Params: {} 2024-03-04T16:13:01.017 DEBUG:boto:Token: None 2024-03-04T16:13:01.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-7/ 2024-03-04T16:13:01.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4= 2024-03-04T16:13:01.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4=', 'Content-Length': '0'} 2024-03-04T16:13:01.018 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000007adf57204e26972c-0065e5f30d-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.019 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:13:01.019 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:13:01.019 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.019 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:13:01.019 DEBUG:boto:Data: 2024-03-04T16:13:01.019 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.019 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.019 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.019 DEBUG:boto:Params: {} 2024-03-04T16:13:01.019 DEBUG:boto:Token: None 2024-03-04T16:13:01.020 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-8/ 2024-03-04T16:13:01.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY= 2024-03-04T16:13:01.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY=', 'Content-Length': '0'} 2024-03-04T16:13:01.021 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000436878e47210e9a4-0065e5f30d-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.021 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:13:01.021 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:13:01.021 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.021 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:13:01.021 DEBUG:boto:Data: 2024-03-04T16:13:01.021 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.021 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.021 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.021 DEBUG:boto:Params: {} 2024-03-04T16:13:01.022 DEBUG:boto:Token: None 2024-03-04T16:13:01.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-9/ 2024-03-04T16:13:01.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c= 2024-03-04T16:13:01.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c=', 'Content-Length': '0'} 2024-03-04T16:13:01.023 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000615e86550411ffb0-0065e5f30d-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.023 DEBUG:boto:path=/sbnvju-7/ 2024-03-04T16:13:01.023 DEBUG:boto:auth_path=/sbnvju-7/ 2024-03-04T16:13:01.023 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.024 DEBUG:boto:Path: /sbnvju-7/ 2024-03-04T16:13:01.024 DEBUG:boto:Data: 2024-03-04T16:13:01.024 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.024 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.024 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.024 DEBUG:boto:Params: {} 2024-03-04T16:13:01.024 DEBUG:boto:Token: None 2024-03-04T16:13:01.024 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-7/ 2024-03-04T16:13:01.024 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4= 2024-03-04T16:13:01.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k916DaykG6Tu1/A4RBfPaAP38U4=', 'Content-Length': '0'} 2024-03-04T16:13:01.026 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000030464c69ed6f99f-0065e5f30d-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.026 DEBUG:boto:path=/sbnvju-8/ 2024-03-04T16:13:01.026 DEBUG:boto:auth_path=/sbnvju-8/ 2024-03-04T16:13:01.026 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.026 DEBUG:boto:Path: /sbnvju-8/ 2024-03-04T16:13:01.026 DEBUG:boto:Data: 2024-03-04T16:13:01.027 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.027 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.027 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.027 DEBUG:boto:Params: {} 2024-03-04T16:13:01.027 DEBUG:boto:Token: None 2024-03-04T16:13:01.027 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-8/ 2024-03-04T16:13:01.027 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY= 2024-03-04T16:13:01.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:La7d8uiETaraofPWY/AQNfY67nY=', 'Content-Length': '0'} 2024-03-04T16:13:01.029 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000f21a4be970763877-0065e5f30d-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.029 DEBUG:boto:path=/sbnvju-9/ 2024-03-04T16:13:01.029 DEBUG:boto:auth_path=/sbnvju-9/ 2024-03-04T16:13:01.029 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.029 DEBUG:boto:Path: /sbnvju-9/ 2024-03-04T16:13:01.029 DEBUG:boto:Data: 2024-03-04T16:13:01.029 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.029 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.029 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.029 DEBUG:boto:Params: {} 2024-03-04T16:13:01.030 DEBUG:boto:Token: None 2024-03-04T16:13:01.030 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-9/ 2024-03-04T16:13:01.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c= 2024-03-04T16:13:01.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PbRe6EusfEgpYuKoNVPzrXdC92c=', 'Content-Length': '0'} 2024-03-04T16:13:01.031 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000984acfcfcdf9e9e1-0065e5f30d-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-03-04T16:13:01.032 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-10 2024-03-04T16:13:01.032 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.032 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.032 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.032 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.032 DEBUG:boto:Data: 2024-03-04T16:13:01.032 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.032 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.032 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.033 DEBUG:boto:Params: {} 2024-03-04T16:13:01.033 DEBUG:boto:Token: None 2024-03-04T16:13:01.033 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.033 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FNUQ/ZomBZ+jPVKY07dg9R8nOzg= 2024-03-04T16:13:01.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FNUQ/ZomBZ+jPVKY07dg9R8nOzg=', 'Content-Length': '0'} 2024-03-04T16:13:01.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ec9af7f1b1c612e-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.050 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-11 2024-03-04T16:13:01.050 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.050 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.050 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.050 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.050 DEBUG:boto:Data: 2024-03-04T16:13:01.050 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.051 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.052 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.052 DEBUG:boto:Params: {} 2024-03-04T16:13:01.052 DEBUG:boto:Token: None 2024-03-04T16:13:01.052 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.052 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JlfyQH3AQn3cSa1irADYYHAHJ8g= 2024-03-04T16:13:01.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JlfyQH3AQn3cSa1irADYYHAHJ8g=', 'Content-Length': '0'} 2024-03-04T16:13:01.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bd2242f53448b7b-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.083 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-12 2024-03-04T16:13:01.083 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.083 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.083 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.083 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.083 DEBUG:boto:Data: 2024-03-04T16:13:01.083 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.084 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.084 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.084 DEBUG:boto:Params: {} 2024-03-04T16:13:01.084 DEBUG:boto:Token: None 2024-03-04T16:13:01.084 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.084 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NSm2yNmS4tTocv1HFNOHDSzyYL0= 2024-03-04T16:13:01.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NSm2yNmS4tTocv1HFNOHDSzyYL0=', 'Content-Length': '0'} 2024-03-04T16:13:01.107 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f45bcdc01cd68648-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.108 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.108 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.108 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.108 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.108 DEBUG:boto:Data: 2024-03-04T16:13:01.108 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.108 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.108 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.109 DEBUG:boto:Params: {} 2024-03-04T16:13:01.109 DEBUG:boto:Token: None 2024-03-04T16:13:01.109 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.109 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a9007c3e19cee25-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.112 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:01.112 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:01.112 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.112 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:01.112 DEBUG:boto:Data: 2024-03-04T16:13:01.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.112 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.112 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.112 DEBUG:boto:Params: {} 2024-03-04T16:13:01.113 DEBUG:boto:Token: None 2024-03-04T16:13:01.113 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/myobj 2024-03-04T16:13:01.113 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JT/eq/k/QWehfxixTsLw/Mi+9zE= 2024-03-04T16:13:01.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JT/eq/k/QWehfxixTsLw/Mi+9zE='} 2024-03-04T16:13:01.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b2d2ddb398753f2-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.122 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.123 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.123 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.123 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.123 DEBUG:boto:Data: 2024-03-04T16:13:01.123 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.123 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.123 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.123 DEBUG:boto:Params: {} 2024-03-04T16:13:01.123 DEBUG:boto:Token: None 2024-03-04T16:13:01.124 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.124 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab25ef9e62981c0f-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.126 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:01.127 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:01.127 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.127 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:01.127 DEBUG:boto:Data: 2024-03-04T16:13:01.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.127 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.127 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.127 DEBUG:boto:Params: {} 2024-03-04T16:13:01.127 DEBUG:boto:Token: None 2024-03-04T16:13:01.128 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/_myobj 2024-03-04T16:13:01.128 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uuj65kjkxCWLmojr/PS24PQBl0w= 2024-03-04T16:13:01.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uuj65kjkxCWLmojr/PS24PQBl0w='} 2024-03-04T16:13:01.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c88c4b17c2fe197d-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.141 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.141 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.141 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.141 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.141 DEBUG:boto:Data: 2024-03-04T16:13:01.141 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.141 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.141 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.141 DEBUG:boto:Params: {} 2024-03-04T16:13:01.142 DEBUG:boto:Token: None 2024-03-04T16:13:01.142 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.142 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e23291d0c33f91ec-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.144 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:01.144 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:01.145 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.145 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:01.145 DEBUG:boto:Data: 2024-03-04T16:13:01.145 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.145 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.145 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.145 DEBUG:boto:Params: {} 2024-03-04T16:13:01.145 DEBUG:boto:Token: None 2024-03-04T16:13:01.145 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/%3A 2024-03-04T16:13:01.146 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KWBdFOafdSQ1aoCAw/cZiNFYIW8= 2024-03-04T16:13:01.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KWBdFOafdSQ1aoCAw/cZiNFYIW8='} 2024-03-04T16:13:01.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000941f0e0fb52ea41a-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.188 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.189 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.190 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.190 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.190 DEBUG:boto:Data: 2024-03-04T16:13:01.190 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.190 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.191 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.191 DEBUG:boto:Params: {} 2024-03-04T16:13:01.191 DEBUG:boto:Token: None 2024-03-04T16:13:01.191 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.191 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e469ed0a3af9e97-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.195 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:01.195 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:01.195 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.195 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:01.195 DEBUG:boto:Data: 2024-03-04T16:13:01.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.195 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.195 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.195 DEBUG:boto:Params: {} 2024-03-04T16:13:01.196 DEBUG:boto:Token: None 2024-03-04T16:13:01.196 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/%26 2024-03-04T16:13:01.196 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eUiuWSRSwc6chH4LR1t2cWPTtPo= 2024-03-04T16:13:01.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eUiuWSRSwc6chH4LR1t2cWPTtPo='} 2024-03-04T16:13:01.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000066d7cfee29ca54e-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.204 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.204 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.204 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.204 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.204 DEBUG:boto:Data: 2024-03-04T16:13:01.204 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.205 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.205 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.205 DEBUG:boto:Params: {} 2024-03-04T16:13:01.205 DEBUG:boto:Token: None 2024-03-04T16:13:01.205 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035a02b4af6749efb-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.207 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:01.207 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:01.207 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.208 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:01.208 DEBUG:boto:Data: 2024-03-04T16:13:01.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.208 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.208 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.208 DEBUG:boto:Params: {} 2024-03-04T16:13:01.208 DEBUG:boto:Token: None 2024-03-04T16:13:01.208 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/. 2024-03-04T16:13:01.208 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UTobUmmktbDIUzwUq0BBcKeVjS4= 2024-03-04T16:13:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UTobUmmktbDIUzwUq0BBcKeVjS4='} 2024-03-04T16:13:01.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbf4a09dc2eab9c7-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.217 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.217 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.217 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.217 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.217 DEBUG:boto:Data: 2024-03-04T16:13:01.217 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.217 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.218 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.218 DEBUG:boto:Params: {} 2024-03-04T16:13:01.218 DEBUG:boto:Token: None 2024-03-04T16:13:01.218 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.218 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9b9ff3f5d651925-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.220 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:01.220 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:01.220 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.220 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:01.221 DEBUG:boto:Data: 2024-03-04T16:13:01.221 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.221 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.221 DEBUG:boto:Params: {} 2024-03-04T16:13:01.221 DEBUG:boto:Token: None 2024-03-04T16:13:01.221 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/.. 2024-03-04T16:13:01.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Rhca47lfTwsjrkCwxrE0Lpqgi60= 2024-03-04T16:13:01.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Rhca47lfTwsjrkCwxrE0Lpqgi60='} 2024-03-04T16:13:01.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000daa927cbc445d27-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.264 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.265 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.265 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.265 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.265 DEBUG:boto:Data: 2024-03-04T16:13:01.265 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.265 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.265 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.265 DEBUG:boto:Params: {} 2024-03-04T16:13:01.265 DEBUG:boto:Token: None 2024-03-04T16:13:01.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb3b4f2058a64451-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.268 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:01.268 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:01.268 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.269 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:01.269 DEBUG:boto:Data: 2024-03-04T16:13:01.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.269 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.269 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.269 DEBUG:boto:Params: {} 2024-03-04T16:13:01.269 DEBUG:boto:Token: None 2024-03-04T16:13:01.269 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/... 2024-03-04T16:13:01.269 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W3+KmyJ26Jq6Yhe5jA+zyVSea5s= 2024-03-04T16:13:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W3+KmyJ26Jq6Yhe5jA+zyVSea5s='} 2024-03-04T16:13:01.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e597700adc1f802b-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.274 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.274 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.274 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.274 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.274 DEBUG:boto:Data: 2024-03-04T16:13:01.275 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.275 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.275 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.275 DEBUG:boto:Params: {} 2024-03-04T16:13:01.275 DEBUG:boto:Token: None 2024-03-04T16:13:01.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082d1a23e16a0f1a6-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.278 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:01.278 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:01.278 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.278 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:01.278 DEBUG:boto:Data: 2024-03-04T16:13:01.278 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.278 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.278 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.278 DEBUG:boto:Params: {} 2024-03-04T16:13:01.278 DEBUG:boto:Token: None 2024-03-04T16:13:01.279 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/.o 2024-03-04T16:13:01.279 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nmh2LhWpPOaY4xEefpQfoZBpA4s= 2024-03-04T16:13:01.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nmh2LhWpPOaY4xEefpQfoZBpA4s='} 2024-03-04T16:13:01.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ed173b176c1a3d8-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.320 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:01.321 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:01.321 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.321 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:01.321 DEBUG:boto:Data: 2024-03-04T16:13:01.321 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.321 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.321 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.321 DEBUG:boto:Params: {} 2024-03-04T16:13:01.321 DEBUG:boto:Token: None 2024-03-04T16:13:01.321 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/ 2024-03-04T16:13:01.322 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M= 2024-03-04T16:13:01.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ihwkJBY6DcipZY01BbtFNjsvA0M=', 'Content-Length': '0'} 2024-03-04T16:13:01.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa8fd15385b40fc8-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.324 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:01.325 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:01.325 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.325 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:01.325 DEBUG:boto:Data: 2024-03-04T16:13:01.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:01.325 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.325 DEBUG:boto:Params: {} 2024-03-04T16:13:01.325 DEBUG:boto:Token: None 2024-03-04T16:13:01.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-10/.o. 2024-03-04T16:13:01.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RC3+fr4wffzsDiSCRZ6XkfZNLcQ= 2024-03-04T16:13:01.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RC3+fr4wffzsDiSCRZ6XkfZNLcQ='} 2024-03-04T16:13:01.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000629b7e2c9284ebc0-0065e5f30d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.331 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.331 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.331 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.331 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.331 DEBUG:boto:Data: 2024-03-04T16:13:01.331 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.331 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.331 DEBUG:boto:Params: {} 2024-03-04T16:13:01.332 DEBUG:boto:Token: None 2024-03-04T16:13:01.332 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.332 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6bfadc5e9585c78-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.334 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:13:01.334 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:13:01.335 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.335 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:13:01.335 DEBUG:boto:Data: 2024-03-04T16:13:01.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.335 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.335 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.335 DEBUG:boto:Params: {} 2024-03-04T16:13:01.335 DEBUG:boto:Token: None 2024-03-04T16:13:01.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/myobj 2024-03-04T16:13:01.335 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OBbdfaoGhXWnIQnQx3oGYH5pTmY= 2024-03-04T16:13:01.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OBbdfaoGhXWnIQnQx3oGYH5pTmY='} 2024-03-04T16:13:01.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008647cce08e9a6fac-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.345 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.345 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.345 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.346 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.346 DEBUG:boto:Data: 2024-03-04T16:13:01.346 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.346 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.346 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.346 DEBUG:boto:Params: {} 2024-03-04T16:13:01.346 DEBUG:boto:Token: None 2024-03-04T16:13:01.346 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.346 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004513321cca33242-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.349 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:13:01.349 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:13:01.349 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.349 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:13:01.349 DEBUG:boto:Data: 2024-03-04T16:13:01.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.349 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.349 DEBUG:boto:Params: {} 2024-03-04T16:13:01.349 DEBUG:boto:Token: None 2024-03-04T16:13:01.350 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/_myobj 2024-03-04T16:13:01.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iAT916wpeyXjkoxhF7ov7l6AP2U= 2024-03-04T16:13:01.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iAT916wpeyXjkoxhF7ov7l6AP2U='} 2024-03-04T16:13:01.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b4ab5a6de983af-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.392 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.393 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.393 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.393 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.393 DEBUG:boto:Data: 2024-03-04T16:13:01.393 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.393 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.393 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.393 DEBUG:boto:Params: {} 2024-03-04T16:13:01.393 DEBUG:boto:Token: None 2024-03-04T16:13:01.394 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050e2d57374ce6fb-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.396 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:13:01.396 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:13:01.396 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.397 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:13:01.397 DEBUG:boto:Data: 2024-03-04T16:13:01.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.397 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.397 DEBUG:boto:Params: {} 2024-03-04T16:13:01.397 DEBUG:boto:Token: None 2024-03-04T16:13:01.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/%3A 2024-03-04T16:13:01.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SHVZOfKLoB21+2J5r0e9GN8Zz3Q= 2024-03-04T16:13:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SHVZOfKLoB21+2J5r0e9GN8Zz3Q='} 2024-03-04T16:13:01.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000776bc9c22d689c76-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.406 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.406 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.406 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.406 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.406 DEBUG:boto:Data: 2024-03-04T16:13:01.406 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.406 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.406 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.406 DEBUG:boto:Params: {} 2024-03-04T16:13:01.407 DEBUG:boto:Token: None 2024-03-04T16:13:01.407 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000589738977dad43c2-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.409 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:13:01.409 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:13:01.409 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.409 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:13:01.409 DEBUG:boto:Data: 2024-03-04T16:13:01.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.410 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.410 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.410 DEBUG:boto:Params: {} 2024-03-04T16:13:01.410 DEBUG:boto:Token: None 2024-03-04T16:13:01.410 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/%26 2024-03-04T16:13:01.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3x0GkvOuW8iVZN+urFt5BlOsQu0= 2024-03-04T16:13:01.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3x0GkvOuW8iVZN+urFt5BlOsQu0='} 2024-03-04T16:13:01.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d3a9cf4d1451a3a-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.457 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.457 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.457 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.457 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.457 DEBUG:boto:Data: 2024-03-04T16:13:01.457 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.457 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.457 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.457 DEBUG:boto:Params: {} 2024-03-04T16:13:01.457 DEBUG:boto:Token: None 2024-03-04T16:13:01.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a662610c703cbc5-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.460 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:13:01.460 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:13:01.460 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.460 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:13:01.460 DEBUG:boto:Data: 2024-03-04T16:13:01.461 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.461 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.461 DEBUG:boto:Params: {} 2024-03-04T16:13:01.461 DEBUG:boto:Token: None 2024-03-04T16:13:01.461 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/. 2024-03-04T16:13:01.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FaZ7qFOjSZbX245gQrCIDsw6zk0= 2024-03-04T16:13:01.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FaZ7qFOjSZbX245gQrCIDsw6zk0='} 2024-03-04T16:13:01.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fdb91ba510c5acd-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.470 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.470 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.470 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.470 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.470 DEBUG:boto:Data: 2024-03-04T16:13:01.470 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.471 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.471 DEBUG:boto:Params: {} 2024-03-04T16:13:01.471 DEBUG:boto:Token: None 2024-03-04T16:13:01.471 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.471 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000819a0a7849e4bf2a-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.473 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:13:01.473 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:13:01.474 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.474 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:13:01.474 DEBUG:boto:Data: 2024-03-04T16:13:01.474 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.474 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.474 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.474 DEBUG:boto:Params: {} 2024-03-04T16:13:01.474 DEBUG:boto:Token: None 2024-03-04T16:13:01.474 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/.. 2024-03-04T16:13:01.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:narjJ/JrlcH2b7c+VfWeYseOB1M= 2024-03-04T16:13:01.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:narjJ/JrlcH2b7c+VfWeYseOB1M='} 2024-03-04T16:13:01.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdbb4adaaf74841c-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.517 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.517 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.517 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.517 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.517 DEBUG:boto:Data: 2024-03-04T16:13:01.517 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.517 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.518 DEBUG:boto:Params: {} 2024-03-04T16:13:01.518 DEBUG:boto:Token: None 2024-03-04T16:13:01.518 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000065e6fcdaa82c215-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.520 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:13:01.520 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:13:01.520 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.520 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:13:01.521 DEBUG:boto:Data: 2024-03-04T16:13:01.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.521 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.521 DEBUG:boto:Params: {} 2024-03-04T16:13:01.521 DEBUG:boto:Token: None 2024-03-04T16:13:01.521 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/... 2024-03-04T16:13:01.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1vVHj+8wJ2IqCHoGODc83Ev6d+w= 2024-03-04T16:13:01.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1vVHj+8wJ2IqCHoGODc83Ev6d+w='} 2024-03-04T16:13:01.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000747fbaba097ae781-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.525 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.525 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.525 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.525 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.525 DEBUG:boto:Data: 2024-03-04T16:13:01.525 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.526 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.526 DEBUG:boto:Params: {} 2024-03-04T16:13:01.526 DEBUG:boto:Token: None 2024-03-04T16:13:01.526 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fd89f840e41594e-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.529 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:13:01.529 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:13:01.529 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.529 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:13:01.529 DEBUG:boto:Data: 2024-03-04T16:13:01.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.529 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.529 DEBUG:boto:Params: {} 2024-03-04T16:13:01.529 DEBUG:boto:Token: None 2024-03-04T16:13:01.530 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/.o 2024-03-04T16:13:01.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m330HOo+7AXxnVfFU5DHg/x13hA= 2024-03-04T16:13:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m330HOo+7AXxnVfFU5DHg/x13hA='} 2024-03-04T16:13:01.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f15252e5623b3082-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.573 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:13:01.573 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:13:01.573 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.573 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:13:01.573 DEBUG:boto:Data: 2024-03-04T16:13:01.573 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.573 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.573 DEBUG:boto:Params: {} 2024-03-04T16:13:01.574 DEBUG:boto:Token: None 2024-03-04T16:13:01.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/ 2024-03-04T16:13:01.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU= 2024-03-04T16:13:01.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VGRCD76oyT0qkgZ45fIqrVhgIZU=', 'Content-Length': '0'} 2024-03-04T16:13:01.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008167f1a298b450c0-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.576 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:13:01.576 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:13:01.576 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.576 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:13:01.576 DEBUG:boto:Data: 2024-03-04T16:13:01.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:01.577 DEBUG:boto:Port: 8000 2024-03-04T16:13:01.577 DEBUG:boto:Params: {} 2024-03-04T16:13:01.577 DEBUG:boto:Token: None 2024-03-04T16:13:01.577 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-11/.o. 2024-03-04T16:13:01.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m5Z0mdhOTfsxXAOknXDbgTyM+CI= 2024-03-04T16:13:01.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m5Z0mdhOTfsxXAOknXDbgTyM+CI='} 2024-03-04T16:13:01.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007120ff9f1659fd3e-0065e5f30d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.582 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.582 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.583 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.583 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.583 DEBUG:boto:Data: 2024-03-04T16:13:01.583 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.583 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.583 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.583 DEBUG:boto:Params: {} 2024-03-04T16:13:01.583 DEBUG:boto:Token: None 2024-03-04T16:13:01.583 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030f5b2ade17565d0-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.586 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:13:01.586 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:13:01.587 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.587 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:13:01.587 DEBUG:boto:Data: 2024-03-04T16:13:01.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.587 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.587 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.588 DEBUG:boto:Params: {} 2024-03-04T16:13:01.588 DEBUG:boto:Token: None 2024-03-04T16:13:01.588 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/myobj 2024-03-04T16:13:01.589 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRSbRgpHQ3uwxgQecd2cGhr4wvg= 2024-03-04T16:13:01.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRSbRgpHQ3uwxgQecd2cGhr4wvg='} 2024-03-04T16:13:01.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008a03fd57efcfeb6-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.601 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.601 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.601 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.601 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.601 DEBUG:boto:Data: 2024-03-04T16:13:01.601 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.601 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.601 DEBUG:boto:Params: {} 2024-03-04T16:13:01.602 DEBUG:boto:Token: None 2024-03-04T16:13:01.602 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000572b4c595a973e61-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.604 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:13:01.604 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:13:01.605 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.605 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:13:01.605 DEBUG:boto:Data: 2024-03-04T16:13:01.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.605 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.605 DEBUG:boto:Params: {} 2024-03-04T16:13:01.605 DEBUG:boto:Token: None 2024-03-04T16:13:01.605 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/_myobj 2024-03-04T16:13:01.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5weUT8OyzPyR5NL5+ONZUSbB+kw= 2024-03-04T16:13:01.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5weUT8OyzPyR5NL5+ONZUSbB+kw='} 2024-03-04T16:13:01.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4992fe6f7cdc3cc-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.652 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.653 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.653 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.653 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.653 DEBUG:boto:Data: 2024-03-04T16:13:01.653 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.653 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.653 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.653 DEBUG:boto:Params: {} 2024-03-04T16:13:01.654 DEBUG:boto:Token: None 2024-03-04T16:13:01.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009108f4f181f0c29e-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.659 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:13:01.659 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:13:01.659 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.660 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:13:01.660 DEBUG:boto:Data: 2024-03-04T16:13:01.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.660 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.660 DEBUG:boto:Params: {} 2024-03-04T16:13:01.660 DEBUG:boto:Token: None 2024-03-04T16:13:01.660 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/%3A 2024-03-04T16:13:01.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rxrRAVHA3xv17/QZXx77clNMAwM= 2024-03-04T16:13:01.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rxrRAVHA3xv17/QZXx77clNMAwM='} 2024-03-04T16:13:01.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e190b805420379f3-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.669 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.669 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.669 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.669 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.669 DEBUG:boto:Data: 2024-03-04T16:13:01.669 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.669 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.669 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.670 DEBUG:boto:Params: {} 2024-03-04T16:13:01.670 DEBUG:boto:Token: None 2024-03-04T16:13:01.670 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.670 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000928942e598eb28b0-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.673 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:13:01.673 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:13:01.673 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.673 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:13:01.673 DEBUG:boto:Data: 2024-03-04T16:13:01.673 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.673 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.673 DEBUG:boto:Params: {} 2024-03-04T16:13:01.674 DEBUG:boto:Token: None 2024-03-04T16:13:01.674 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/%26 2024-03-04T16:13:01.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e2NYiXpE6FhYZH2OC4NRQjsNJkI= 2024-03-04T16:13:01.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e2NYiXpE6FhYZH2OC4NRQjsNJkI='} 2024-03-04T16:13:01.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ab8b3b0fd5843b1-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.716 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.717 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.717 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.717 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.717 DEBUG:boto:Data: 2024-03-04T16:13:01.717 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.717 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.717 DEBUG:boto:Params: {} 2024-03-04T16:13:01.717 DEBUG:boto:Token: None 2024-03-04T16:13:01.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f0441d4a4ed7539-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.720 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:13:01.720 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:13:01.720 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.720 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:13:01.720 DEBUG:boto:Data: 2024-03-04T16:13:01.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.720 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.721 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.721 DEBUG:boto:Params: {} 2024-03-04T16:13:01.721 DEBUG:boto:Token: None 2024-03-04T16:13:01.721 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/. 2024-03-04T16:13:01.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/2aPlfc6sdxdHo60uguO+h3IlmA= 2024-03-04T16:13:01.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/2aPlfc6sdxdHo60uguO+h3IlmA='} 2024-03-04T16:13:01.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053d6363b59ad7c4a-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.731 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.731 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.732 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.732 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.732 DEBUG:boto:Data: 2024-03-04T16:13:01.732 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.732 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.732 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.732 DEBUG:boto:Params: {} 2024-03-04T16:13:01.732 DEBUG:boto:Token: None 2024-03-04T16:13:01.733 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.733 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abd3f76722e1f636-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.735 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:13:01.735 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:13:01.735 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.735 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:13:01.736 DEBUG:boto:Data: 2024-03-04T16:13:01.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.736 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.736 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.736 DEBUG:boto:Params: {} 2024-03-04T16:13:01.736 DEBUG:boto:Token: None 2024-03-04T16:13:01.736 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/.. 2024-03-04T16:13:01.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oKAQXG/+lvXnvZcMPQSNmQa7p5Q= 2024-03-04T16:13:01.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oKAQXG/+lvXnvZcMPQSNmQa7p5Q='} 2024-03-04T16:13:01.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000588820773ffb3fe-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.741 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.741 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.741 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.741 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.741 DEBUG:boto:Data: 2024-03-04T16:13:01.742 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.742 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.742 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.742 DEBUG:boto:Params: {} 2024-03-04T16:13:01.742 DEBUG:boto:Token: None 2024-03-04T16:13:01.742 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.742 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001352d79b7a473aeb-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.745 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:13:01.745 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:13:01.746 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.746 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:13:01.746 DEBUG:boto:Data: 2024-03-04T16:13:01.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.746 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.746 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.746 DEBUG:boto:Params: {} 2024-03-04T16:13:01.746 DEBUG:boto:Token: None 2024-03-04T16:13:01.746 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/... 2024-03-04T16:13:01.747 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ykJrchw2CRaWXxMGR7vu+wu1uEs= 2024-03-04T16:13:01.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ykJrchw2CRaWXxMGR7vu+wu1uEs='} 2024-03-04T16:13:01.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042ce787cf7581641-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.797 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.797 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.797 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.797 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.797 DEBUG:boto:Data: 2024-03-04T16:13:01.797 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.797 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.797 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.797 DEBUG:boto:Params: {} 2024-03-04T16:13:01.798 DEBUG:boto:Token: None 2024-03-04T16:13:01.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bb2c72c7d99b66c-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.800 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:13:01.801 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:13:01.801 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.801 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:13:01.801 DEBUG:boto:Data: 2024-03-04T16:13:01.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.801 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.801 DEBUG:boto:Params: {} 2024-03-04T16:13:01.801 DEBUG:boto:Token: None 2024-03-04T16:13:01.802 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/.o 2024-03-04T16:13:01.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HuZgPxkbeAFnA/PbSjmyiZHEXyc= 2024-03-04T16:13:01.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HuZgPxkbeAFnA/PbSjmyiZHEXyc='} 2024-03-04T16:13:01.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a56a724f4949c5c-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.805 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:13:01.805 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:13:01.806 DEBUG:boto:Method: HEAD 2024-03-04T16:13:01.806 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:13:01.806 DEBUG:boto:Data: 2024-03-04T16:13:01.806 DEBUG:boto:Headers: {} 2024-03-04T16:13:01.806 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.806 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.806 DEBUG:boto:Params: {} 2024-03-04T16:13:01.806 DEBUG:boto:Token: None 2024-03-04T16:13:01.806 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/ 2024-03-04T16:13:01.807 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4= 2024-03-04T16:13:01.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/od+ZB1AYfzP5OzvPoF4P48p4q4=', 'Content-Length': '0'} 2024-03-04T16:13:01.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13203f7d07fd0a2-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.809 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:13:01.809 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:13:01.809 DEBUG:boto:Method: PUT 2024-03-04T16:13:01.809 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:13:01.809 DEBUG:boto:Data: 2024-03-04T16:13:01.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:13:01.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:01.809 DEBUG:boto:Port: 8001 2024-03-04T16:13:01.809 DEBUG:boto:Params: {} 2024-03-04T16:13:01.810 DEBUG:boto:Token: None 2024-03-04T16:13:01.810 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:13:01 GMT /sbnvju-12/.o. 2024-03-04T16:13:01.810 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2iOHM3M06tiNwTih9wFRT10ZPe8= 2024-03-04T16:13:01.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:13:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2iOHM3M06tiNwTih9wFRT10ZPe8='} 2024-03-04T16:13:01.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075e7db85ba5cc7eb-0065e5f30d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:01.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:01.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:01.853 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:13:01.899 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:01.899 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:01.945 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:13:01.964 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709568781.097665_52.1","last_update":"2024-03-04T16:13:01.097665Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:13:01.964 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709568781.097665_52.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:01.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:13:01.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:01.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:01.965 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:13:02.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:02.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:02.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:02.055+0000 7fb7df212800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:02.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:02.056+0000 7fb7df212800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:02.075 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568781.097665_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:13:01.097665Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:02Z"} 2024-03-04T16:13:02.075 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568781.097665_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:13:01.097665Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:02Z"} 2024-03-04T16:13:02.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568781.097665_52.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:13:01.097665Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:13:02.076 DEBUG:rgw_multi.tests:log_status={0: '1_1709568781.097665_52.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:02.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568781.097665_52.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:02.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:13:02.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:13:02.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:02.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:13:02.076 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:13:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:02.172+0000 7f890560f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:02.173+0000 7f890560f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:13:02.193 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568781.097665_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:13:01.097665Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:02Z"} 2024-03-04T16:13:02.193 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709568781.097665_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:13:01.097665Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:13:02Z"} 2024-03-04T16:13:02.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709568781.097665_52.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:13:01.097665Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:13:02.194 DEBUG:rgw_multi.tests:log_status={0: '1_1709568781.097665_52.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:02.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1709568781.097665_52.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:13:02.194 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:13:02.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:02.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:02.194 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:13:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:02.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:02.293+0000 7f79ce89d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:07.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:07.294+0000 7f79ce89d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:12.294+0000 7f79ce89d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:17.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:17.296+0000 7f79ce89d800 1 bucket sync caught up with source: 2024-03-04T16:13:17.297 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.18.6, , 00000000001.16.6, , 00000000001.15.6, 00000000002.14.6, 00000000001.16.6, 00000000001.12.6, 00000000001.20.6, ] 2024-03-04T16:13:17.297 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.18.6, , 00000000001.16.6, , 00000000001.15.6, 00000000002.14.6, 00000000001.16.6, 00000000001.12.6, 00000000001.20.6, ] 2024-03-04T16:13:17.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:13:17.296+0000 7f79ce89d800 0 bucket checkpoint complete 2024-03-04T16:13:17.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-10 zones={test-zone2, test-zone1} 2024-03-04T16:13:17.319 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.320 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.320 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.320 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:17.320 DEBUG:boto:Data: 2024-03-04T16:13:17.320 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.320 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.320 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.320 DEBUG:boto:Params: {} 2024-03-04T16:13:17.321 DEBUG:boto:Token: None 2024-03-04T16:13:17.321 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.321 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:COk8hQRmn9rOQoGUZL5+GIEB/pU= 2024-03-04T16:13:17.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:COk8hQRmn9rOQoGUZL5+GIEB/pU=', 'Content-Length': '0'} 2024-03-04T16:13:17.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095fc2ebbe6d21bbb-0065e5f31d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.324 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.324 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.324 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.324 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:17.324 DEBUG:boto:Data: 2024-03-04T16:13:17.324 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.324 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.324 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.324 DEBUG:boto:Params: {} 2024-03-04T16:13:17.325 DEBUG:boto:Token: None 2024-03-04T16:13:17.325 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.325 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:COk8hQRmn9rOQoGUZL5+GIEB/pU= 2024-03-04T16:13:17.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:COk8hQRmn9rOQoGUZL5+GIEB/pU=', 'Content-Length': '0'} 2024-03-04T16:13:17.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfcc56bd21772f44-0065e5f31d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:13:17.328 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.328 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.328 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.328 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.328 DEBUG:boto:Method: GET 2024-03-04T16:13:17.329 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.329 DEBUG:boto:Data: 2024-03-04T16:13:17.329 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.329 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.329 DEBUG:boto:Params: {} 2024-03-04T16:13:17.329 DEBUG:boto:Token: None 2024-03-04T16:13:17.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/?versions 2024-03-04T16:13:17.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs= 2024-03-04T16:13:17.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs=', 'Content-Length': '0'} 2024-03-04T16:13:17.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000823d08aae117188-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.332 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:13:17.335 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:13:17.336 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:13:17.336 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.336 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.336 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.336 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.336 DEBUG:boto:Method: GET 2024-03-04T16:13:17.336 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.336 DEBUG:boto:Data: 2024-03-04T16:13:17.337 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.337 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.337 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.337 DEBUG:boto:Params: {} 2024-03-04T16:13:17.337 DEBUG:boto:Token: None 2024-03-04T16:13:17.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/?versions 2024-03-04T16:13:17.337 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs= 2024-03-04T16:13:17.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs=', 'Content-Length': '0'} 2024-03-04T16:13:17.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc74256ea8be11b4-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.340 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:13:17.341 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:13:17.342 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:13:17.342 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.342 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.342 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.342 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.342 DEBUG:boto:Method: GET 2024-03-04T16:13:17.342 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.342 DEBUG:boto:Data: 2024-03-04T16:13:17.342 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.342 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.342 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.342 DEBUG:boto:Params: {} 2024-03-04T16:13:17.343 DEBUG:boto:Token: None 2024-03-04T16:13:17.343 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/?versions 2024-03-04T16:13:17.343 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs= 2024-03-04T16:13:17.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs=', 'Content-Length': '0'} 2024-03-04T16:13:17.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f0c3f441e01fe6-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.346 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.347 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.347 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.347 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.347 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.347 DEBUG:boto:Method: GET 2024-03-04T16:13:17.347 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:17.347 DEBUG:boto:Data: 2024-03-04T16:13:17.347 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.347 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.347 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.347 DEBUG:boto:Params: {} 2024-03-04T16:13:17.348 DEBUG:boto:Token: None 2024-03-04T16:13:17.348 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/?versions 2024-03-04T16:13:17.348 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs= 2024-03-04T16:13:17.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XB/FROEMXbjg95oDdhdoKdjJ0Xs=', 'Content-Length': '0'} 2024-03-04T16:13:17.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5446c41692f968e-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.351 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:17.351 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.352 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.352 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.352 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.352 DEBUG:boto:Method: GET 2024-03-04T16:13:17.352 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.352 DEBUG:boto:Data: 2024-03-04T16:13:17.352 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.352 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.353 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.353 DEBUG:boto:Params: {} 2024-03-04T16:13:17.353 DEBUG:boto:Token: None 2024-03-04T16:13:17.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw= 2024-03-04T16:13:17.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw=', 'Content-Length': '0'} 2024-03-04T16:13:17.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b80e3b687c866fb0-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.396 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.397 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.397 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.398 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.398 DEBUG:boto:Method: GET 2024-03-04T16:13:17.398 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.398 DEBUG:boto:Data: 2024-03-04T16:13:17.399 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.399 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.399 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.399 DEBUG:boto:Params: {} 2024-03-04T16:13:17.399 DEBUG:boto:Token: None 2024-03-04T16:13:17.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.400 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw= 2024-03-04T16:13:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw=', 'Content-Length': '0'} 2024-03-04T16:13:17.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d9fea1ce507ea03-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.402 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.402 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.402 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.402 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.402 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.402 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.402 DEBUG:boto:Data: 2024-03-04T16:13:17.402 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.402 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.403 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.403 DEBUG:boto:Params: {} 2024-03-04T16:13:17.404 DEBUG:boto:Token: None 2024-03-04T16:13:17.404 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.404 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5u2UhaWT2dyffqRqLwbtIG40yFQ= 2024-03-04T16:13:17.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5u2UhaWT2dyffqRqLwbtIG40yFQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dafa3be0d8fce324-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.406 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.406 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.406 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.406 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.406 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.407 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.407 DEBUG:boto:Data: 2024-03-04T16:13:17.407 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.407 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.407 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.407 DEBUG:boto:Params: {} 2024-03-04T16:13:17.408 DEBUG:boto:Token: None 2024-03-04T16:13:17.408 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5u2UhaWT2dyffqRqLwbtIG40yFQ= 2024-03-04T16:13:17.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5u2UhaWT2dyffqRqLwbtIG40yFQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e83e45b9ffb2161f-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:17.411 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.411 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.411 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.412 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.412 DEBUG:boto:Method: GET 2024-03-04T16:13:17.412 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.412 DEBUG:boto:Data: 2024-03-04T16:13:17.412 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.412 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.412 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.412 DEBUG:boto:Params: {} 2024-03-04T16:13:17.412 DEBUG:boto:Token: None 2024-03-04T16:13:17.412 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw= 2024-03-04T16:13:17.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw=', 'Content-Length': '0'} 2024-03-04T16:13:17.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4ed033e952eb256-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.460 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.460 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.460 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.460 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.461 DEBUG:boto:Method: GET 2024-03-04T16:13:17.461 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.461 DEBUG:boto:Data: 2024-03-04T16:13:17.461 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.461 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.461 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.461 DEBUG:boto:Params: {} 2024-03-04T16:13:17.461 DEBUG:boto:Token: None 2024-03-04T16:13:17.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:17.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw= 2024-03-04T16:13:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1NVvhpWcAjizPCAdlLznxSzXHDw=', 'Content-Length': '0'} 2024-03-04T16:13:17.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dcda55f9912b071-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.464 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.464 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.464 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.464 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.464 DEBUG:boto:Method: GET 2024-03-04T16:13:17.464 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:17.464 DEBUG:boto:Data: 2024-03-04T16:13:17.465 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.465 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.465 DEBUG:boto:Params: {} 2024-03-04T16:13:17.465 DEBUG:boto:Token: None 2024-03-04T16:13:17.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e5ae02da704177-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.468 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.469 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.469 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.469 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.469 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.469 DEBUG:boto:Method: GET 2024-03-04T16:13:17.469 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:17.470 DEBUG:boto:Data: 2024-03-04T16:13:17.470 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.471 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.471 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.471 DEBUG:boto:Params: {} 2024-03-04T16:13:17.471 DEBUG:boto:Token: None 2024-03-04T16:13:17.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.472 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079914adafd4f793e-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.474 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.475 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.475 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.475 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.475 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:17.475 DEBUG:boto:Data: 2024-03-04T16:13:17.475 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.475 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.475 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.476 DEBUG:boto:Params: {} 2024-03-04T16:13:17.476 DEBUG:boto:Token: None 2024-03-04T16:13:17.476 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26 2024-03-04T16:13:17.476 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OAOrJq/WZ48hlHkOL5TBqty7Xyc= 2024-03-04T16:13:17.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OAOrJq/WZ48hlHkOL5TBqty7Xyc=', 'Content-Length': '0'} 2024-03-04T16:13:17.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c41e49b0308167f-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.478 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.478 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.478 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.478 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:17.478 DEBUG:boto:Data: 2024-03-04T16:13:17.479 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.479 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.479 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.479 DEBUG:boto:Params: {} 2024-03-04T16:13:17.479 DEBUG:boto:Token: None 2024-03-04T16:13:17.479 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26 2024-03-04T16:13:17.479 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OAOrJq/WZ48hlHkOL5TBqty7Xyc= 2024-03-04T16:13:17.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OAOrJq/WZ48hlHkOL5TBqty7Xyc=', 'Content-Length': '0'} 2024-03-04T16:13:17.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc8d20c3f893375f-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:17.481 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.482 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.482 DEBUG:boto:Method: GET 2024-03-04T16:13:17.482 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:17.482 DEBUG:boto:Data: 2024-03-04T16:13:17.482 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.482 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.482 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.482 DEBUG:boto:Params: {} 2024-03-04T16:13:17.482 DEBUG:boto:Token: None 2024-03-04T16:13:17.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26 2024-03-04T16:13:17.483 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PBWM1fl9dnOhmpXzoJzr+MPLsSg= 2024-03-04T16:13:17.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PBWM1fl9dnOhmpXzoJzr+MPLsSg=', 'Content-Length': '0'} 2024-03-04T16:13:17.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbb81b08191049c3-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.484 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:17.485 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:17.485 DEBUG:boto:Method: GET 2024-03-04T16:13:17.485 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:17.485 DEBUG:boto:Data: 2024-03-04T16:13:17.485 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.485 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.485 DEBUG:boto:Params: {} 2024-03-04T16:13:17.485 DEBUG:boto:Token: None 2024-03-04T16:13:17.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/%26 2024-03-04T16:13:17.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PBWM1fl9dnOhmpXzoJzr+MPLsSg= 2024-03-04T16:13:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PBWM1fl9dnOhmpXzoJzr+MPLsSg=', 'Content-Length': '0'} 2024-03-04T16:13:17.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c8617545cb4616d-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.528 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.528 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.528 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.529 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.529 DEBUG:boto:Method: GET 2024-03-04T16:13:17.529 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:17.529 DEBUG:boto:Data: 2024-03-04T16:13:17.529 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.529 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.529 DEBUG:boto:Params: {} 2024-03-04T16:13:17.529 DEBUG:boto:Token: None 2024-03-04T16:13:17.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000078b70d3890b2106-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.532 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.533 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.533 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.533 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.533 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:17.533 DEBUG:boto:Method: GET 2024-03-04T16:13:17.533 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:17.533 DEBUG:boto:Data: 2024-03-04T16:13:17.533 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.533 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.534 DEBUG:boto:Params: {} 2024-03-04T16:13:17.534 DEBUG:boto:Token: None 2024-03-04T16:13:17.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac407310c598adc-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.537 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:17.538 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.538 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.538 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.538 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.538 DEBUG:boto:Method: GET 2024-03-04T16:13:17.538 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.538 DEBUG:boto:Data: 2024-03-04T16:13:17.538 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.538 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.539 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.539 DEBUG:boto:Params: {} 2024-03-04T16:13:17.539 DEBUG:boto:Token: None 2024-03-04T16:13:17.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA= 2024-03-04T16:13:17.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA=', 'Content-Length': '0'} 2024-03-04T16:13:17.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5c891d76ed80a2-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.588 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.588 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.588 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.588 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.589 DEBUG:boto:Method: GET 2024-03-04T16:13:17.589 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.589 DEBUG:boto:Data: 2024-03-04T16:13:17.589 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.589 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.589 DEBUG:boto:Params: {} 2024-03-04T16:13:17.589 DEBUG:boto:Token: None 2024-03-04T16:13:17.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA= 2024-03-04T16:13:17.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA=', 'Content-Length': '0'} 2024-03-04T16:13:17.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab07ccc3ee0c3947-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.592 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.593 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.593 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.593 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.593 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.593 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.593 DEBUG:boto:Data: 2024-03-04T16:13:17.593 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.593 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.593 DEBUG:boto:Params: {} 2024-03-04T16:13:17.593 DEBUG:boto:Token: None 2024-03-04T16:13:17.594 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HMqruGMD4Z/EdZnBOrZWLuu6rzg= 2024-03-04T16:13:17.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HMqruGMD4Z/EdZnBOrZWLuu6rzg=', 'Content-Length': '0'} 2024-03-04T16:13:17.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fdb8e0b8f2795cc-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.596 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.596 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.596 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.596 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.597 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.597 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.597 DEBUG:boto:Data: 2024-03-04T16:13:17.597 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.597 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.597 DEBUG:boto:Params: {} 2024-03-04T16:13:17.597 DEBUG:boto:Token: None 2024-03-04T16:13:17.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HMqruGMD4Z/EdZnBOrZWLuu6rzg= 2024-03-04T16:13:17.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HMqruGMD4Z/EdZnBOrZWLuu6rzg=', 'Content-Length': '0'} 2024-03-04T16:13:17.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039a561104a7a3eb2-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:17.599 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.600 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.600 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.600 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.600 DEBUG:boto:Method: GET 2024-03-04T16:13:17.600 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.600 DEBUG:boto:Data: 2024-03-04T16:13:17.600 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.600 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.600 DEBUG:boto:Params: {} 2024-03-04T16:13:17.600 DEBUG:boto:Token: None 2024-03-04T16:13:17.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA= 2024-03-04T16:13:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA=', 'Content-Length': '0'} 2024-03-04T16:13:17.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e003c99ab156779-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.603 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.603 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.603 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.603 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:17.603 DEBUG:boto:Method: GET 2024-03-04T16:13:17.603 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.604 DEBUG:boto:Data: 2024-03-04T16:13:17.604 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.604 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.604 DEBUG:boto:Params: {} 2024-03-04T16:13:17.604 DEBUG:boto:Token: None 2024-03-04T16:13:17.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:17.604 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA= 2024-03-04T16:13:17.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0rPu49fv3HXLn9IFs9SqHu+PjQA=', 'Content-Length': '0'} 2024-03-04T16:13:17.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efe27d06bd763f92-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.607 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.607 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.607 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.607 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.607 DEBUG:boto:Method: GET 2024-03-04T16:13:17.607 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:17.607 DEBUG:boto:Data: 2024-03-04T16:13:17.607 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.607 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.608 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.608 DEBUG:boto:Params: {} 2024-03-04T16:13:17.608 DEBUG:boto:Token: None 2024-03-04T16:13:17.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d810dc4915c370c-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.611 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.611 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.611 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.611 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.611 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.612 DEBUG:boto:Method: GET 2024-03-04T16:13:17.612 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:17.612 DEBUG:boto:Data: 2024-03-04T16:13:17.612 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.612 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.612 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.612 DEBUG:boto:Params: {} 2024-03-04T16:13:17.612 DEBUG:boto:Token: None 2024-03-04T16:13:17.612 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.612 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2837cfd62fba4ab-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.615 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.616 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.616 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.616 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.617 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:17.617 DEBUG:boto:Data: 2024-03-04T16:13:17.617 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.617 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.617 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.617 DEBUG:boto:Params: {} 2024-03-04T16:13:17.617 DEBUG:boto:Token: None 2024-03-04T16:13:17.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/. 2024-03-04T16:13:17.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xOSocP8LOMv7XUMxU9vazYoT0BY= 2024-03-04T16:13:17.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xOSocP8LOMv7XUMxU9vazYoT0BY=', 'Content-Length': '0'} 2024-03-04T16:13:17.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef6986663454502-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.619 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.620 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.620 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.620 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:17.620 DEBUG:boto:Data: 2024-03-04T16:13:17.620 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.620 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.620 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.620 DEBUG:boto:Params: {} 2024-03-04T16:13:17.620 DEBUG:boto:Token: None 2024-03-04T16:13:17.620 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/. 2024-03-04T16:13:17.621 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xOSocP8LOMv7XUMxU9vazYoT0BY= 2024-03-04T16:13:17.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xOSocP8LOMv7XUMxU9vazYoT0BY=', 'Content-Length': '0'} 2024-03-04T16:13:17.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2d3814f0c52262a-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:17.623 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.623 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.623 DEBUG:boto:Method: GET 2024-03-04T16:13:17.623 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:17.623 DEBUG:boto:Data: 2024-03-04T16:13:17.623 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.623 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.623 DEBUG:boto:Params: {} 2024-03-04T16:13:17.623 DEBUG:boto:Token: None 2024-03-04T16:13:17.624 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/. 2024-03-04T16:13:17.624 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T6/DNCpuA8UY9/7qzG/Y0BAPWFU= 2024-03-04T16:13:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T6/DNCpuA8UY9/7qzG/Y0BAPWFU=', 'Content-Length': '0'} 2024-03-04T16:13:17.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000857f444cb6bf3816-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.626 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:17.626 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:17.626 DEBUG:boto:Method: GET 2024-03-04T16:13:17.626 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:17.626 DEBUG:boto:Data: 2024-03-04T16:13:17.626 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.627 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.627 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.627 DEBUG:boto:Params: {} 2024-03-04T16:13:17.627 DEBUG:boto:Token: None 2024-03-04T16:13:17.627 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/. 2024-03-04T16:13:17.627 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T6/DNCpuA8UY9/7qzG/Y0BAPWFU= 2024-03-04T16:13:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T6/DNCpuA8UY9/7qzG/Y0BAPWFU=', 'Content-Length': '0'} 2024-03-04T16:13:17.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005469e8ab98ae6141-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.629 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.629 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.629 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.629 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.630 DEBUG:boto:Method: GET 2024-03-04T16:13:17.630 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:17.630 DEBUG:boto:Data: 2024-03-04T16:13:17.630 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.630 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.630 DEBUG:boto:Params: {} 2024-03-04T16:13:17.630 DEBUG:boto:Token: None 2024-03-04T16:13:17.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1621efefc766c0-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.633 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.634 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.634 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.634 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.634 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:17.634 DEBUG:boto:Method: GET 2024-03-04T16:13:17.634 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:17.634 DEBUG:boto:Data: 2024-03-04T16:13:17.634 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.634 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.635 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.635 DEBUG:boto:Params: {} 2024-03-04T16:13:17.635 DEBUG:boto:Token: None 2024-03-04T16:13:17.635 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.635 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806b5bfc0f1cda1c-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.637 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:17.638 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.638 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.638 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.638 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.638 DEBUG:boto:Method: GET 2024-03-04T16:13:17.639 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.639 DEBUG:boto:Data: 2024-03-04T16:13:17.639 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.639 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.639 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.639 DEBUG:boto:Params: {} 2024-03-04T16:13:17.639 DEBUG:boto:Token: None 2024-03-04T16:13:17.639 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.639 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4= 2024-03-04T16:13:17.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4=', 'Content-Length': '0'} 2024-03-04T16:13:17.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021b0a7ab22dd9e48-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.642 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.642 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.642 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.642 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.642 DEBUG:boto:Method: GET 2024-03-04T16:13:17.642 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.642 DEBUG:boto:Data: 2024-03-04T16:13:17.642 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.642 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.642 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.643 DEBUG:boto:Params: {} 2024-03-04T16:13:17.643 DEBUG:boto:Token: None 2024-03-04T16:13:17.643 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.643 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4= 2024-03-04T16:13:17.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4=', 'Content-Length': '0'} 2024-03-04T16:13:17.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000056aa7d924b8d6ee-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.692 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.692 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.692 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.692 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.693 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.693 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.693 DEBUG:boto:Data: 2024-03-04T16:13:17.693 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.693 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.693 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.693 DEBUG:boto:Params: {} 2024-03-04T16:13:17.693 DEBUG:boto:Token: None 2024-03-04T16:13:17.693 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LlbxfJyDqmfCau4cPfharPI2dIQ= 2024-03-04T16:13:17.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LlbxfJyDqmfCau4cPfharPI2dIQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f971086271faa5f8-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.696 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.696 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.696 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.696 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.696 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.697 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.697 DEBUG:boto:Data: 2024-03-04T16:13:17.697 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.697 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.697 DEBUG:boto:Params: {} 2024-03-04T16:13:17.697 DEBUG:boto:Token: None 2024-03-04T16:13:17.697 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.697 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LlbxfJyDqmfCau4cPfharPI2dIQ= 2024-03-04T16:13:17.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LlbxfJyDqmfCau4cPfharPI2dIQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075fc6ac9c59dfda8-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:17.700 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.700 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.700 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.700 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.700 DEBUG:boto:Method: GET 2024-03-04T16:13:17.700 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.701 DEBUG:boto:Data: 2024-03-04T16:13:17.701 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.701 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.701 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.701 DEBUG:boto:Params: {} 2024-03-04T16:13:17.701 DEBUG:boto:Token: None 2024-03-04T16:13:17.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.701 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4= 2024-03-04T16:13:17.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4=', 'Content-Length': '0'} 2024-03-04T16:13:17.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000552ac7e602a9d870-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.703 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.703 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.703 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.703 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:17.704 DEBUG:boto:Method: GET 2024-03-04T16:13:17.704 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.704 DEBUG:boto:Data: 2024-03-04T16:13:17.704 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.704 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.704 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.704 DEBUG:boto:Params: {} 2024-03-04T16:13:17.704 DEBUG:boto:Token: None 2024-03-04T16:13:17.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:17.705 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4= 2024-03-04T16:13:17.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k/VQ8m5EC6uo7CIQ8igMlJHZLb4=', 'Content-Length': '0'} 2024-03-04T16:13:17.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007714fb0aa888f492-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.707 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.707 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.707 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.707 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.707 DEBUG:boto:Method: GET 2024-03-04T16:13:17.707 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:17.707 DEBUG:boto:Data: 2024-03-04T16:13:17.707 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.707 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.707 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.707 DEBUG:boto:Params: {} 2024-03-04T16:13:17.708 DEBUG:boto:Token: None 2024-03-04T16:13:17.708 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.708 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f22c4b58abca89-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.711 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.712 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.712 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.713 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.713 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.713 DEBUG:boto:Method: GET 2024-03-04T16:13:17.713 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:17.713 DEBUG:boto:Data: 2024-03-04T16:13:17.713 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.713 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.714 DEBUG:boto:Params: {} 2024-03-04T16:13:17.714 DEBUG:boto:Token: None 2024-03-04T16:13:17.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0912cf13437531-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.717 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.717 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.718 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.718 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.718 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:17.718 DEBUG:boto:Data: 2024-03-04T16:13:17.718 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.718 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.718 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.718 DEBUG:boto:Params: {} 2024-03-04T16:13:17.718 DEBUG:boto:Token: None 2024-03-04T16:13:17.718 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.. 2024-03-04T16:13:17.719 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENK30lSsu25923/pAWPglTjwBgI= 2024-03-04T16:13:17.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENK30lSsu25923/pAWPglTjwBgI=', 'Content-Length': '0'} 2024-03-04T16:13:17.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a50a3dba728efde3-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.720 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.721 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.721 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.721 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:17.721 DEBUG:boto:Data: 2024-03-04T16:13:17.721 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.721 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.721 DEBUG:boto:Params: {} 2024-03-04T16:13:17.721 DEBUG:boto:Token: None 2024-03-04T16:13:17.721 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.. 2024-03-04T16:13:17.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENK30lSsu25923/pAWPglTjwBgI= 2024-03-04T16:13:17.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENK30lSsu25923/pAWPglTjwBgI=', 'Content-Length': '0'} 2024-03-04T16:13:17.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a181c00614c74696-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:17.724 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.724 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.724 DEBUG:boto:Method: GET 2024-03-04T16:13:17.725 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:17.725 DEBUG:boto:Data: 2024-03-04T16:13:17.725 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.725 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.725 DEBUG:boto:Params: {} 2024-03-04T16:13:17.725 DEBUG:boto:Token: None 2024-03-04T16:13:17.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.. 2024-03-04T16:13:17.725 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+htIAODRIqIp9tRa5gaZn1Ud0+U= 2024-03-04T16:13:17.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+htIAODRIqIp9tRa5gaZn1Ud0+U=', 'Content-Length': '0'} 2024-03-04T16:13:17.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000addb13815c52efb-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.772 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:17.772 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:17.772 DEBUG:boto:Method: GET 2024-03-04T16:13:17.773 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:17.773 DEBUG:boto:Data: 2024-03-04T16:13:17.773 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.773 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.773 DEBUG:boto:Params: {} 2024-03-04T16:13:17.773 DEBUG:boto:Token: None 2024-03-04T16:13:17.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.. 2024-03-04T16:13:17.773 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+htIAODRIqIp9tRa5gaZn1Ud0+U= 2024-03-04T16:13:17.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+htIAODRIqIp9tRa5gaZn1Ud0+U=', 'Content-Length': '0'} 2024-03-04T16:13:17.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e5c7eee1bb32a6c-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.776 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.776 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.776 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.776 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.776 DEBUG:boto:Method: GET 2024-03-04T16:13:17.776 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:17.776 DEBUG:boto:Data: 2024-03-04T16:13:17.776 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.776 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.776 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.777 DEBUG:boto:Params: {} 2024-03-04T16:13:17.777 DEBUG:boto:Token: None 2024-03-04T16:13:17.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb91e920828f1e8-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.779 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.780 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.780 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.780 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.780 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:17.780 DEBUG:boto:Method: GET 2024-03-04T16:13:17.780 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:17.781 DEBUG:boto:Data: 2024-03-04T16:13:17.781 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.781 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.781 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.781 DEBUG:boto:Params: {} 2024-03-04T16:13:17.781 DEBUG:boto:Token: None 2024-03-04T16:13:17.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.781 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37e0ab181eb44d2-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.784 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:17.784 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.784 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.784 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.785 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.785 DEBUG:boto:Method: GET 2024-03-04T16:13:17.785 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.785 DEBUG:boto:Data: 2024-03-04T16:13:17.785 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.785 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.785 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.785 DEBUG:boto:Params: {} 2024-03-04T16:13:17.785 DEBUG:boto:Token: None 2024-03-04T16:13:17.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ= 2024-03-04T16:13:17.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009734ec354a2e0e9b-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.788 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.788 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.788 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.788 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.788 DEBUG:boto:Method: GET 2024-03-04T16:13:17.788 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.788 DEBUG:boto:Data: 2024-03-04T16:13:17.788 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.788 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.789 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.789 DEBUG:boto:Params: {} 2024-03-04T16:13:17.789 DEBUG:boto:Token: None 2024-03-04T16:13:17.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ= 2024-03-04T16:13:17.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f61fc9b9696e8e0-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.791 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.791 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.791 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.791 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.791 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.792 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.792 DEBUG:boto:Data: 2024-03-04T16:13:17.792 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.792 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.792 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.792 DEBUG:boto:Params: {} 2024-03-04T16:13:17.792 DEBUG:boto:Token: None 2024-03-04T16:13:17.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo11u1dX/9biCHRlxvjYIz/+Mu4= 2024-03-04T16:13:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo11u1dX/9biCHRlxvjYIz/+Mu4=', 'Content-Length': '0'} 2024-03-04T16:13:17.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032a0d3a96ecda2b9-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.794 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.794 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.794 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.794 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.795 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.795 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.795 DEBUG:boto:Data: 2024-03-04T16:13:17.795 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.795 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.795 DEBUG:boto:Params: {} 2024-03-04T16:13:17.795 DEBUG:boto:Token: None 2024-03-04T16:13:17.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo11u1dX/9biCHRlxvjYIz/+Mu4= 2024-03-04T16:13:17.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo11u1dX/9biCHRlxvjYIz/+Mu4=', 'Content-Length': '0'} 2024-03-04T16:13:17.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3ac43da76882854-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:17.798 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.798 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.798 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.798 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.798 DEBUG:boto:Method: GET 2024-03-04T16:13:17.798 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.798 DEBUG:boto:Data: 2024-03-04T16:13:17.798 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.798 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.798 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.798 DEBUG:boto:Params: {} 2024-03-04T16:13:17.799 DEBUG:boto:Token: None 2024-03-04T16:13:17.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ= 2024-03-04T16:13:17.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022a9ff9502edc6ec-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.801 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.801 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.801 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.801 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:17.801 DEBUG:boto:Method: GET 2024-03-04T16:13:17.801 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.802 DEBUG:boto:Data: 2024-03-04T16:13:17.802 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.802 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.802 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.802 DEBUG:boto:Params: {} 2024-03-04T16:13:17.802 DEBUG:boto:Token: None 2024-03-04T16:13:17.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:17.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ= 2024-03-04T16:13:17.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9g8gf4jfY8B+hpvvj+acphGXmiQ=', 'Content-Length': '0'} 2024-03-04T16:13:17.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b755c87fe48c250-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.805 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.805 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.805 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.805 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.805 DEBUG:boto:Method: GET 2024-03-04T16:13:17.805 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:17.805 DEBUG:boto:Data: 2024-03-04T16:13:17.806 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.806 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.806 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.806 DEBUG:boto:Params: {} 2024-03-04T16:13:17.806 DEBUG:boto:Token: None 2024-03-04T16:13:17.806 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.806 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c8b6a6497f7feb-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.809 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.810 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.810 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.810 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.810 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.810 DEBUG:boto:Method: GET 2024-03-04T16:13:17.810 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:17.810 DEBUG:boto:Data: 2024-03-04T16:13:17.810 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.810 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.810 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.811 DEBUG:boto:Params: {} 2024-03-04T16:13:17.811 DEBUG:boto:Token: None 2024-03-04T16:13:17.811 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.811 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088bf1babd26919e1-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.860 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.861 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.861 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.861 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.861 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:17.861 DEBUG:boto:Data: 2024-03-04T16:13:17.861 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.861 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.861 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.861 DEBUG:boto:Params: {} 2024-03-04T16:13:17.861 DEBUG:boto:Token: None 2024-03-04T16:13:17.862 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/... 2024-03-04T16:13:17.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lNJK9vpMSi1wK35ASMIxxARhfQ4= 2024-03-04T16:13:17.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lNJK9vpMSi1wK35ASMIxxARhfQ4=', 'Content-Length': '0'} 2024-03-04T16:13:17.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de7f3c5aecad7804-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.864 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.864 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.864 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.865 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:17.865 DEBUG:boto:Data: 2024-03-04T16:13:17.865 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.865 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.865 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.865 DEBUG:boto:Params: {} 2024-03-04T16:13:17.865 DEBUG:boto:Token: None 2024-03-04T16:13:17.865 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/... 2024-03-04T16:13:17.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lNJK9vpMSi1wK35ASMIxxARhfQ4= 2024-03-04T16:13:17.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lNJK9vpMSi1wK35ASMIxxARhfQ4=', 'Content-Length': '0'} 2024-03-04T16:13:17.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c99055921f85d171-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:17.869 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.869 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.869 DEBUG:boto:Method: GET 2024-03-04T16:13:17.869 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:17.869 DEBUG:boto:Data: 2024-03-04T16:13:17.869 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.869 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.869 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.870 DEBUG:boto:Params: {} 2024-03-04T16:13:17.870 DEBUG:boto:Token: None 2024-03-04T16:13:17.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/... 2024-03-04T16:13:17.870 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UiMpdzsPkcDEnM8nyVRlDZzB5TA= 2024-03-04T16:13:17.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UiMpdzsPkcDEnM8nyVRlDZzB5TA=', 'Content-Length': '0'} 2024-03-04T16:13:17.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000394fbba45e122922-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.920 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:17.920 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:17.920 DEBUG:boto:Method: GET 2024-03-04T16:13:17.920 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:17.921 DEBUG:boto:Data: 2024-03-04T16:13:17.921 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.921 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.921 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.921 DEBUG:boto:Params: {} 2024-03-04T16:13:17.921 DEBUG:boto:Token: None 2024-03-04T16:13:17.921 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/... 2024-03-04T16:13:17.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UiMpdzsPkcDEnM8nyVRlDZzB5TA= 2024-03-04T16:13:17.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UiMpdzsPkcDEnM8nyVRlDZzB5TA=', 'Content-Length': '0'} 2024-03-04T16:13:17.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032d7fe7d39aa803a-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.924 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.924 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.924 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.924 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.924 DEBUG:boto:Method: GET 2024-03-04T16:13:17.925 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:17.925 DEBUG:boto:Data: 2024-03-04T16:13:17.925 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.925 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.925 DEBUG:boto:Params: {} 2024-03-04T16:13:17.925 DEBUG:boto:Token: None 2024-03-04T16:13:17.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.925 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308781fe5eb18424-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.928 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.929 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.929 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.929 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.929 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:17.929 DEBUG:boto:Method: GET 2024-03-04T16:13:17.929 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:17.929 DEBUG:boto:Data: 2024-03-04T16:13:17.929 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.929 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.930 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.930 DEBUG:boto:Params: {} 2024-03-04T16:13:17.930 DEBUG:boto:Token: None 2024-03-04T16:13:17.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ac59faabde8f5b-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.936 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:17.937 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.937 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.937 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.937 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.937 DEBUG:boto:Method: GET 2024-03-04T16:13:17.937 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.937 DEBUG:boto:Data: 2024-03-04T16:13:17.937 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.937 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.937 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.937 DEBUG:boto:Params: {} 2024-03-04T16:13:17.938 DEBUG:boto:Token: None 2024-03-04T16:13:17.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo= 2024-03-04T16:13:17.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo=', 'Content-Length': '0'} 2024-03-04T16:13:17.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffda4fd0abf6ddfd-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.940 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.940 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.940 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.940 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.940 DEBUG:boto:Method: GET 2024-03-04T16:13:17.940 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.940 DEBUG:boto:Data: 2024-03-04T16:13:17.940 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.941 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.941 DEBUG:boto:Params: {} 2024-03-04T16:13:17.941 DEBUG:boto:Token: None 2024-03-04T16:13:17.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.941 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo= 2024-03-04T16:13:17.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo=', 'Content-Length': '0'} 2024-03-04T16:13:17.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfda1939aff71ccf-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.944 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.944 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.944 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.944 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.944 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.944 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.944 DEBUG:boto:Data: 2024-03-04T16:13:17.944 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.945 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.945 DEBUG:boto:Params: {} 2024-03-04T16:13:17.945 DEBUG:boto:Token: None 2024-03-04T16:13:17.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qQ1+/KSeFBbd9Q/+n/zZgVcmTJA= 2024-03-04T16:13:17.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qQ1+/KSeFBbd9Q/+n/zZgVcmTJA=', 'Content-Length': '0'} 2024-03-04T16:13:17.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed167a6a092765f7-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.947 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.947 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.947 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.947 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.948 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.948 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.948 DEBUG:boto:Data: 2024-03-04T16:13:17.948 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.948 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.948 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.948 DEBUG:boto:Params: {} 2024-03-04T16:13:17.948 DEBUG:boto:Token: None 2024-03-04T16:13:17.948 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qQ1+/KSeFBbd9Q/+n/zZgVcmTJA= 2024-03-04T16:13:17.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qQ1+/KSeFBbd9Q/+n/zZgVcmTJA=', 'Content-Length': '0'} 2024-03-04T16:13:17.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000333c029c04af4cf1-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:17.951 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.951 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.951 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.951 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.951 DEBUG:boto:Method: GET 2024-03-04T16:13:17.951 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.951 DEBUG:boto:Data: 2024-03-04T16:13:17.952 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.952 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.952 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.952 DEBUG:boto:Params: {} 2024-03-04T16:13:17.952 DEBUG:boto:Token: None 2024-03-04T16:13:17.952 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.952 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo= 2024-03-04T16:13:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo=', 'Content-Length': '0'} 2024-03-04T16:13:17.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017862ba7751faeb2-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.954 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.955 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.955 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.955 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.955 DEBUG:boto:Method: GET 2024-03-04T16:13:17.955 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.955 DEBUG:boto:Data: 2024-03-04T16:13:17.955 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.955 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.955 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.955 DEBUG:boto:Params: {} 2024-03-04T16:13:17.955 DEBUG:boto:Token: None 2024-03-04T16:13:17.956 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:17.956 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo= 2024-03-04T16:13:17.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rIwCLsKJBk3VHh/Yr/nOodlCiWo=', 'Content-Length': '0'} 2024-03-04T16:13:17.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4b865c11961f359-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.959 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.959 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.959 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.959 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.960 DEBUG:boto:Method: GET 2024-03-04T16:13:17.960 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:17.960 DEBUG:boto:Data: 2024-03-04T16:13:17.960 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.960 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.960 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.960 DEBUG:boto:Params: {} 2024-03-04T16:13:17.960 DEBUG:boto:Token: None 2024-03-04T16:13:17.960 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.960 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031ead5de195322ab-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.963 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.963 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.964 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.964 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.964 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.964 DEBUG:boto:Method: GET 2024-03-04T16:13:17.964 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:17.964 DEBUG:boto:Data: 2024-03-04T16:13:17.964 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.964 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.964 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.964 DEBUG:boto:Params: {} 2024-03-04T16:13:17.964 DEBUG:boto:Token: None 2024-03-04T16:13:17.965 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.965 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1857a86be4b955e-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.967 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:17.968 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.968 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.968 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.968 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:17.968 DEBUG:boto:Data: 2024-03-04T16:13:17.968 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.968 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.968 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.969 DEBUG:boto:Params: {} 2024-03-04T16:13:17.969 DEBUG:boto:Token: None 2024-03-04T16:13:17.969 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o 2024-03-04T16:13:17.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4wPmG5qeSxsfMuDCiTWjvkLHuhE= 2024-03-04T16:13:17.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4wPmG5qeSxsfMuDCiTWjvkLHuhE=', 'Content-Length': '0'} 2024-03-04T16:13:17.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c15f456a2612a57a-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.971 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.971 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.971 DEBUG:boto:Method: HEAD 2024-03-04T16:13:17.971 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:17.971 DEBUG:boto:Data: 2024-03-04T16:13:17.971 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.971 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.971 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.971 DEBUG:boto:Params: {} 2024-03-04T16:13:17.971 DEBUG:boto:Token: None 2024-03-04T16:13:17.972 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o 2024-03-04T16:13:17.972 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4wPmG5qeSxsfMuDCiTWjvkLHuhE= 2024-03-04T16:13:17.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4wPmG5qeSxsfMuDCiTWjvkLHuhE=', 'Content-Length': '0'} 2024-03-04T16:13:17.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bb044e054348119-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:17.974 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.974 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.974 DEBUG:boto:Method: GET 2024-03-04T16:13:17.974 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:17.974 DEBUG:boto:Data: 2024-03-04T16:13:17.974 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.974 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.974 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.974 DEBUG:boto:Params: {} 2024-03-04T16:13:17.975 DEBUG:boto:Token: None 2024-03-04T16:13:17.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o 2024-03-04T16:13:17.975 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5HKWhHOFQHbc40x6EjN39kZNuqc= 2024-03-04T16:13:17.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5HKWhHOFQHbc40x6EjN39kZNuqc=', 'Content-Length': '0'} 2024-03-04T16:13:17.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4b05221d5029ee4-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.977 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:17.977 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:17.977 DEBUG:boto:Method: GET 2024-03-04T16:13:17.977 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:17.977 DEBUG:boto:Data: 2024-03-04T16:13:17.977 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:17.977 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.978 DEBUG:boto:Params: {} 2024-03-04T16:13:17.978 DEBUG:boto:Token: None 2024-03-04T16:13:17.978 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/.o 2024-03-04T16:13:17.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5HKWhHOFQHbc40x6EjN39kZNuqc= 2024-03-04T16:13:17.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5HKWhHOFQHbc40x6EjN39kZNuqc=', 'Content-Length': '0'} 2024-03-04T16:13:17.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6e14827450e2cc2-0065e5f31d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:17.980 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:17.980 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:17.980 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.980 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:17.980 DEBUG:boto:Method: GET 2024-03-04T16:13:17.980 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:17.980 DEBUG:boto:Data: 2024-03-04T16:13:17.980 DEBUG:boto:Headers: {} 2024-03-04T16:13:17.980 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:17.980 DEBUG:boto:Port: 8000 2024-03-04T16:13:17.981 DEBUG:boto:Params: {} 2024-03-04T16:13:17.981 DEBUG:boto:Token: None 2024-03-04T16:13:17.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:17 GMT /sbnvju-10/ 2024-03-04T16:13:17.981 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM= 2024-03-04T16:13:17.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S1m9TGDWH3uA+yEPmpvmXxJkFtM=', 'Content-Length': '0'} 2024-03-04T16:13:17.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c86ee126adcab4-0065e5f31d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.028 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.029 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.029 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.029 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:18.029 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:18.029 DEBUG:boto:Method: GET 2024-03-04T16:13:18.029 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:18.029 DEBUG:boto:Data: 2024-03-04T16:13:18.029 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.029 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.029 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.029 DEBUG:boto:Params: {} 2024-03-04T16:13:18.030 DEBUG:boto:Token: None 2024-03-04T16:13:18.030 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000930c7edde68aa88c-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.032 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:18.033 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.033 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.033 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.033 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.033 DEBUG:boto:Method: GET 2024-03-04T16:13:18.033 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.033 DEBUG:boto:Data: 2024-03-04T16:13:18.034 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.034 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.034 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.034 DEBUG:boto:Params: {} 2024-03-04T16:13:18.034 DEBUG:boto:Token: None 2024-03-04T16:13:18.034 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE= 2024-03-04T16:13:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE=', 'Content-Length': '0'} 2024-03-04T16:13:18.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d631cc73662d526-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.036 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.037 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.037 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.037 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.037 DEBUG:boto:Method: GET 2024-03-04T16:13:18.037 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.037 DEBUG:boto:Data: 2024-03-04T16:13:18.037 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.038 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.038 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.038 DEBUG:boto:Params: {} 2024-03-04T16:13:18.038 DEBUG:boto:Token: None 2024-03-04T16:13:18.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE= 2024-03-04T16:13:18.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE=', 'Content-Length': '0'} 2024-03-04T16:13:18.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003eacd2188d7849d0-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.080 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.080 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.080 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.081 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.081 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.081 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.081 DEBUG:boto:Data: 2024-03-04T16:13:18.081 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.081 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.081 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.081 DEBUG:boto:Params: {} 2024-03-04T16:13:18.081 DEBUG:boto:Token: None 2024-03-04T16:13:18.082 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RPlFo4dt5TAFJ3mOVO0NyKZSi1c= 2024-03-04T16:13:18.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RPlFo4dt5TAFJ3mOVO0NyKZSi1c=', 'Content-Length': '0'} 2024-03-04T16:13:18.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b34a019eb254bb9-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.084 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.084 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.084 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.084 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.084 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.084 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.085 DEBUG:boto:Data: 2024-03-04T16:13:18.085 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.085 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.085 DEBUG:boto:Params: {} 2024-03-04T16:13:18.085 DEBUG:boto:Token: None 2024-03-04T16:13:18.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RPlFo4dt5TAFJ3mOVO0NyKZSi1c= 2024-03-04T16:13:18.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RPlFo4dt5TAFJ3mOVO0NyKZSi1c=', 'Content-Length': '0'} 2024-03-04T16:13:18.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf3b5f2f48e23749-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:18.088 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.088 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.088 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.088 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.088 DEBUG:boto:Method: GET 2024-03-04T16:13:18.088 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.088 DEBUG:boto:Data: 2024-03-04T16:13:18.088 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.088 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.089 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.089 DEBUG:boto:Params: {} 2024-03-04T16:13:18.089 DEBUG:boto:Token: None 2024-03-04T16:13:18.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE= 2024-03-04T16:13:18.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE=', 'Content-Length': '0'} 2024-03-04T16:13:18.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a28fedd8a4533e04-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.092 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.092 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.092 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.092 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.092 DEBUG:boto:Method: GET 2024-03-04T16:13:18.092 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.092 DEBUG:boto:Data: 2024-03-04T16:13:18.093 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.093 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.093 DEBUG:boto:Params: {} 2024-03-04T16:13:18.093 DEBUG:boto:Token: None 2024-03-04T16:13:18.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:18.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE= 2024-03-04T16:13:18.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BTtTryHXVvkf9yBFcLfXAJ3qawE=', 'Content-Length': '0'} 2024-03-04T16:13:18.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093601e53c95022fe-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.095 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.095 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.095 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.095 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.095 DEBUG:boto:Method: GET 2024-03-04T16:13:18.096 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.096 DEBUG:boto:Data: 2024-03-04T16:13:18.096 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.096 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.096 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.096 DEBUG:boto:Params: {} 2024-03-04T16:13:18.096 DEBUG:boto:Token: None 2024-03-04T16:13:18.096 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.096 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a40ae0dafca3fd-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.099 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.099 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.100 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.100 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.100 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.100 DEBUG:boto:Method: GET 2024-03-04T16:13:18.100 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.100 DEBUG:boto:Data: 2024-03-04T16:13:18.100 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.100 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.101 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.101 DEBUG:boto:Params: {} 2024-03-04T16:13:18.101 DEBUG:boto:Token: None 2024-03-04T16:13:18.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.101 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009162974f79efcc26-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.103 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.104 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.104 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.104 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.104 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:18.104 DEBUG:boto:Data: 2024-03-04T16:13:18.104 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.104 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.104 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.104 DEBUG:boto:Params: {} 2024-03-04T16:13:18.105 DEBUG:boto:Token: None 2024-03-04T16:13:18.105 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o. 2024-03-04T16:13:18.105 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2mu4plJ4FAvZns6hzxkhT8RdGog= 2024-03-04T16:13:18.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2mu4plJ4FAvZns6hzxkhT8RdGog=', 'Content-Length': '0'} 2024-03-04T16:13:18.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097b9ac7c5ef1f8a9-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.107 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.107 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.107 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.108 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:18.108 DEBUG:boto:Data: 2024-03-04T16:13:18.108 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.108 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.108 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.108 DEBUG:boto:Params: {} 2024-03-04T16:13:18.108 DEBUG:boto:Token: None 2024-03-04T16:13:18.108 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o. 2024-03-04T16:13:18.109 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2mu4plJ4FAvZns6hzxkhT8RdGog= 2024-03-04T16:13:18.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2mu4plJ4FAvZns6hzxkhT8RdGog=', 'Content-Length': '0'} 2024-03-04T16:13:18.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e601123308ae20a-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:18.111 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.111 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.111 DEBUG:boto:Method: GET 2024-03-04T16:13:18.111 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:18.111 DEBUG:boto:Data: 2024-03-04T16:13:18.111 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.111 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.111 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.112 DEBUG:boto:Params: {} 2024-03-04T16:13:18.112 DEBUG:boto:Token: None 2024-03-04T16:13:18.112 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o. 2024-03-04T16:13:18.112 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+DMT/7PBOimfXws93cNlyX/7isc= 2024-03-04T16:13:18.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+DMT/7PBOimfXws93cNlyX/7isc=', 'Content-Length': '0'} 2024-03-04T16:13:18.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001802eee6f1199e0d-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.115 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:18.115 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:18.115 DEBUG:boto:Method: GET 2024-03-04T16:13:18.115 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:18.115 DEBUG:boto:Data: 2024-03-04T16:13:18.115 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.115 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.115 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.115 DEBUG:boto:Params: {} 2024-03-04T16:13:18.115 DEBUG:boto:Token: None 2024-03-04T16:13:18.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.o. 2024-03-04T16:13:18.116 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+DMT/7PBOimfXws93cNlyX/7isc= 2024-03-04T16:13:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+DMT/7PBOimfXws93cNlyX/7isc=', 'Content-Length': '0'} 2024-03-04T16:13:18.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000815eab4e5708bec9-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.118 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.119 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.119 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.119 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.119 DEBUG:boto:Method: GET 2024-03-04T16:13:18.119 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.119 DEBUG:boto:Data: 2024-03-04T16:13:18.119 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.119 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.119 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.120 DEBUG:boto:Params: {} 2024-03-04T16:13:18.120 DEBUG:boto:Token: None 2024-03-04T16:13:18.120 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0d1d7fc5e62932-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.123 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.123 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.123 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.123 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.123 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.124 DEBUG:boto:Method: GET 2024-03-04T16:13:18.124 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:18.124 DEBUG:boto:Data: 2024-03-04T16:13:18.124 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.124 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.124 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.124 DEBUG:boto:Params: {} 2024-03-04T16:13:18.124 DEBUG:boto:Token: None 2024-03-04T16:13:18.124 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.124 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f350ddaa36d21e80-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.127 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:18.127 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.128 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.128 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.128 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.128 DEBUG:boto:Method: GET 2024-03-04T16:13:18.128 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.128 DEBUG:boto:Data: 2024-03-04T16:13:18.128 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.128 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.128 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.128 DEBUG:boto:Params: {} 2024-03-04T16:13:18.128 DEBUG:boto:Token: None 2024-03-04T16:13:18.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug= 2024-03-04T16:13:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug=', 'Content-Length': '0'} 2024-03-04T16:13:18.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026101e1dc012775d-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.131 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.131 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.131 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.132 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.132 DEBUG:boto:Method: GET 2024-03-04T16:13:18.132 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.132 DEBUG:boto:Data: 2024-03-04T16:13:18.132 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.132 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.132 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.132 DEBUG:boto:Params: {} 2024-03-04T16:13:18.133 DEBUG:boto:Token: None 2024-03-04T16:13:18.133 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug= 2024-03-04T16:13:18.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug=', 'Content-Length': '0'} 2024-03-04T16:13:18.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7a9dddfe3a67beb-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.135 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.135 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.135 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.135 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.135 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.135 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.135 DEBUG:boto:Data: 2024-03-04T16:13:18.135 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.135 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.135 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.135 DEBUG:boto:Params: {} 2024-03-04T16:13:18.136 DEBUG:boto:Token: None 2024-03-04T16:13:18.136 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.136 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/mFbf/K+U9q4hrySkxyuGPrf0g= 2024-03-04T16:13:18.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/mFbf/K+U9q4hrySkxyuGPrf0g=', 'Content-Length': '0'} 2024-03-04T16:13:18.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e728b1fd24dba960-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.138 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.138 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.138 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.138 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.138 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.138 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.138 DEBUG:boto:Data: 2024-03-04T16:13:18.138 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.139 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.139 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.139 DEBUG:boto:Params: {} 2024-03-04T16:13:18.139 DEBUG:boto:Token: None 2024-03-04T16:13:18.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.139 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/mFbf/K+U9q4hrySkxyuGPrf0g= 2024-03-04T16:13:18.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/mFbf/K+U9q4hrySkxyuGPrf0g=', 'Content-Length': '0'} 2024-03-04T16:13:18.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005af04bbe4039c471-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:18.141 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.141 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.142 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.142 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.142 DEBUG:boto:Method: GET 2024-03-04T16:13:18.142 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.142 DEBUG:boto:Data: 2024-03-04T16:13:18.142 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.142 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.142 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.142 DEBUG:boto:Params: {} 2024-03-04T16:13:18.143 DEBUG:boto:Token: None 2024-03-04T16:13:18.143 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.143 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug= 2024-03-04T16:13:18.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug=', 'Content-Length': '0'} 2024-03-04T16:13:18.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000867c4542f75fa3db-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.145 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.145 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.145 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.146 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.146 DEBUG:boto:Method: GET 2024-03-04T16:13:18.146 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.146 DEBUG:boto:Data: 2024-03-04T16:13:18.146 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.146 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.146 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.146 DEBUG:boto:Params: {} 2024-03-04T16:13:18.146 DEBUG:boto:Token: None 2024-03-04T16:13:18.147 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:18.147 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug= 2024-03-04T16:13:18.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KKS0ZeU+YoqbZ/WFL/cLWaR0Gug=', 'Content-Length': '0'} 2024-03-04T16:13:18.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dec21d5fb92812bb-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.148 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.149 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.149 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.149 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.149 DEBUG:boto:Method: GET 2024-03-04T16:13:18.149 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.150 DEBUG:boto:Data: 2024-03-04T16:13:18.150 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.150 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.150 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.150 DEBUG:boto:Params: {} 2024-03-04T16:13:18.150 DEBUG:boto:Token: None 2024-03-04T16:13:18.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3078029f700ca2-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.153 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.153 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.153 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.153 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.154 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.154 DEBUG:boto:Method: GET 2024-03-04T16:13:18.154 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.154 DEBUG:boto:Data: 2024-03-04T16:13:18.154 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.154 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.154 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.154 DEBUG:boto:Params: {} 2024-03-04T16:13:18.154 DEBUG:boto:Token: None 2024-03-04T16:13:18.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.155 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e615473a659d02e7-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.204 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.205 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.205 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.205 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.205 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:18.205 DEBUG:boto:Data: 2024-03-04T16:13:18.205 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.205 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.205 DEBUG:boto:Params: {} 2024-03-04T16:13:18.206 DEBUG:boto:Token: None 2024-03-04T16:13:18.206 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A 2024-03-04T16:13:18.206 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zP5HLVgHRBlHsvLD3bB2K3swwFY= 2024-03-04T16:13:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zP5HLVgHRBlHsvLD3bB2K3swwFY=', 'Content-Length': '0'} 2024-03-04T16:13:18.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad81c78890a1461e-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.208 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.208 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.209 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.209 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:18.209 DEBUG:boto:Data: 2024-03-04T16:13:18.209 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.209 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.209 DEBUG:boto:Params: {} 2024-03-04T16:13:18.209 DEBUG:boto:Token: None 2024-03-04T16:13:18.209 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A 2024-03-04T16:13:18.209 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zP5HLVgHRBlHsvLD3bB2K3swwFY= 2024-03-04T16:13:18.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zP5HLVgHRBlHsvLD3bB2K3swwFY=', 'Content-Length': '0'} 2024-03-04T16:13:18.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040c525381b09b338-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:18.211 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.211 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.212 DEBUG:boto:Method: GET 2024-03-04T16:13:18.212 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:18.212 DEBUG:boto:Data: 2024-03-04T16:13:18.212 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.212 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.212 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.212 DEBUG:boto:Params: {} 2024-03-04T16:13:18.212 DEBUG:boto:Token: None 2024-03-04T16:13:18.212 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A 2024-03-04T16:13:18.212 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YtcKGrEM9keSqdYFULYvjCy+NJo= 2024-03-04T16:13:18.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YtcKGrEM9keSqdYFULYvjCy+NJo=', 'Content-Length': '0'} 2024-03-04T16:13:18.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983ce278297f8808-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.256 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:18.256 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:18.257 DEBUG:boto:Method: GET 2024-03-04T16:13:18.257 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:18.257 DEBUG:boto:Data: 2024-03-04T16:13:18.257 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.257 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.257 DEBUG:boto:Params: {} 2024-03-04T16:13:18.257 DEBUG:boto:Token: None 2024-03-04T16:13:18.257 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%3A 2024-03-04T16:13:18.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YtcKGrEM9keSqdYFULYvjCy+NJo= 2024-03-04T16:13:18.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YtcKGrEM9keSqdYFULYvjCy+NJo=', 'Content-Length': '0'} 2024-03-04T16:13:18.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bb2880ddd3ce94b-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.260 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.260 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.260 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.260 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.260 DEBUG:boto:Method: GET 2024-03-04T16:13:18.260 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.260 DEBUG:boto:Data: 2024-03-04T16:13:18.260 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.260 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.261 DEBUG:boto:Params: {} 2024-03-04T16:13:18.263 DEBUG:boto:Token: None 2024-03-04T16:13:18.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff9cbbf497808c26-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.267 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.267 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.267 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.267 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.267 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.267 DEBUG:boto:Method: GET 2024-03-04T16:13:18.267 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:18.267 DEBUG:boto:Data: 2024-03-04T16:13:18.268 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.268 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.268 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.268 DEBUG:boto:Params: {} 2024-03-04T16:13:18.268 DEBUG:boto:Token: None 2024-03-04T16:13:18.268 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.268 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4ed855c581effe-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.271 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:18.271 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.271 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.272 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.272 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.272 DEBUG:boto:Method: GET 2024-03-04T16:13:18.272 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.272 DEBUG:boto:Data: 2024-03-04T16:13:18.272 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.272 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.272 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.272 DEBUG:boto:Params: {} 2024-03-04T16:13:18.272 DEBUG:boto:Token: None 2024-03-04T16:13:18.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI= 2024-03-04T16:13:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI=', 'Content-Length': '0'} 2024-03-04T16:13:18.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f60f05b4d04713d9-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.276 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.276 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.276 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.276 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.276 DEBUG:boto:Method: GET 2024-03-04T16:13:18.277 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.277 DEBUG:boto:Data: 2024-03-04T16:13:18.277 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.277 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.277 DEBUG:boto:Params: {} 2024-03-04T16:13:18.277 DEBUG:boto:Token: None 2024-03-04T16:13:18.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI= 2024-03-04T16:13:18.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI=', 'Content-Length': '0'} 2024-03-04T16:13:18.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb1864e830b343df-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.279 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.280 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.280 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.280 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.280 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.280 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.280 DEBUG:boto:Data: 2024-03-04T16:13:18.280 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.280 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.280 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.280 DEBUG:boto:Params: {} 2024-03-04T16:13:18.280 DEBUG:boto:Token: None 2024-03-04T16:13:18.281 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vTlwTyKR/Iz1Petd5KHTvSUwAQw= 2024-03-04T16:13:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vTlwTyKR/Iz1Petd5KHTvSUwAQw=', 'Content-Length': '0'} 2024-03-04T16:13:18.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd7c5d8ee9d70e7b-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.284 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.284 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.284 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.284 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.285 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.285 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.285 DEBUG:boto:Data: 2024-03-04T16:13:18.285 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.285 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.285 DEBUG:boto:Params: {} 2024-03-04T16:13:18.286 DEBUG:boto:Token: None 2024-03-04T16:13:18.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vTlwTyKR/Iz1Petd5KHTvSUwAQw= 2024-03-04T16:13:18.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vTlwTyKR/Iz1Petd5KHTvSUwAQw=', 'Content-Length': '0'} 2024-03-04T16:13:18.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c189d28a7896d0e5-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:18.289 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.290 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.290 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.290 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.290 DEBUG:boto:Method: GET 2024-03-04T16:13:18.290 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.290 DEBUG:boto:Data: 2024-03-04T16:13:18.291 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.291 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.291 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.291 DEBUG:boto:Params: {} 2024-03-04T16:13:18.291 DEBUG:boto:Token: None 2024-03-04T16:13:18.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.292 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI= 2024-03-04T16:13:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI=', 'Content-Length': '0'} 2024-03-04T16:13:18.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058b7991d986dbc9f-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.295 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.295 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.295 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.295 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.295 DEBUG:boto:Method: GET 2024-03-04T16:13:18.296 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.296 DEBUG:boto:Data: 2024-03-04T16:13:18.296 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.296 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.296 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.296 DEBUG:boto:Params: {} 2024-03-04T16:13:18.296 DEBUG:boto:Token: None 2024-03-04T16:13:18.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:18.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI= 2024-03-04T16:13:18.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AiH03zbu30AiSPRxhb8y2jp3/fI=', 'Content-Length': '0'} 2024-03-04T16:13:18.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb9d7e9853516f37-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.300 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.300 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.300 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.300 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.300 DEBUG:boto:Method: GET 2024-03-04T16:13:18.300 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.301 DEBUG:boto:Data: 2024-03-04T16:13:18.301 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.301 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.301 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.301 DEBUG:boto:Params: {} 2024-03-04T16:13:18.301 DEBUG:boto:Token: None 2024-03-04T16:13:18.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46354c5391afbef-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.305 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.305 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.306 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.306 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.306 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.306 DEBUG:boto:Method: GET 2024-03-04T16:13:18.306 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.306 DEBUG:boto:Data: 2024-03-04T16:13:18.306 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.306 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.306 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.306 DEBUG:boto:Params: {} 2024-03-04T16:13:18.306 DEBUG:boto:Token: None 2024-03-04T16:13:18.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c6598e412b8eb7-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.310 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.310 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.310 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.310 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.310 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:18.310 DEBUG:boto:Data: 2024-03-04T16:13:18.310 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.311 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.311 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.311 DEBUG:boto:Params: {} 2024-03-04T16:13:18.311 DEBUG:boto:Token: None 2024-03-04T16:13:18.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj 2024-03-04T16:13:18.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LynT2fu5YCKL30frKwA/Bg5u75A= 2024-03-04T16:13:18.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LynT2fu5YCKL30frKwA/Bg5u75A=', 'Content-Length': '0'} 2024-03-04T16:13:18.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cd226823e73f660-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.313 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.313 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.314 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.314 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:18.314 DEBUG:boto:Data: 2024-03-04T16:13:18.314 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.314 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.314 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.314 DEBUG:boto:Params: {} 2024-03-04T16:13:18.315 DEBUG:boto:Token: None 2024-03-04T16:13:18.315 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj 2024-03-04T16:13:18.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LynT2fu5YCKL30frKwA/Bg5u75A= 2024-03-04T16:13:18.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LynT2fu5YCKL30frKwA/Bg5u75A=', 'Content-Length': '0'} 2024-03-04T16:13:18.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5b4e0c77551c754-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:18.317 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.317 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.317 DEBUG:boto:Method: GET 2024-03-04T16:13:18.317 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:18.318 DEBUG:boto:Data: 2024-03-04T16:13:18.318 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.318 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.318 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.318 DEBUG:boto:Params: {} 2024-03-04T16:13:18.318 DEBUG:boto:Token: None 2024-03-04T16:13:18.318 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj 2024-03-04T16:13:18.318 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3AVzEEJJIK1zdYoW5Uu69/dIgEE= 2024-03-04T16:13:18.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3AVzEEJJIK1zdYoW5Uu69/dIgEE=', 'Content-Length': '0'} 2024-03-04T16:13:18.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000017701f25265a91a-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.368 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:18.368 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:18.369 DEBUG:boto:Method: GET 2024-03-04T16:13:18.369 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:18.369 DEBUG:boto:Data: 2024-03-04T16:13:18.369 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.369 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.369 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.369 DEBUG:boto:Params: {} 2024-03-04T16:13:18.369 DEBUG:boto:Token: None 2024-03-04T16:13:18.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/_myobj 2024-03-04T16:13:18.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3AVzEEJJIK1zdYoW5Uu69/dIgEE= 2024-03-04T16:13:18.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3AVzEEJJIK1zdYoW5Uu69/dIgEE=', 'Content-Length': '0'} 2024-03-04T16:13:18.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdf55255f20e9ee5-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.372 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.372 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.372 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.372 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.372 DEBUG:boto:Method: GET 2024-03-04T16:13:18.372 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.372 DEBUG:boto:Data: 2024-03-04T16:13:18.373 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.373 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.373 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.373 DEBUG:boto:Params: {} 2024-03-04T16:13:18.373 DEBUG:boto:Token: None 2024-03-04T16:13:18.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208078a5a2e034f9-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.376 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.376 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.376 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.377 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.377 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.377 DEBUG:boto:Method: GET 2024-03-04T16:13:18.377 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:18.377 DEBUG:boto:Data: 2024-03-04T16:13:18.377 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.377 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.378 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.378 DEBUG:boto:Params: {} 2024-03-04T16:13:18.378 DEBUG:boto:Token: None 2024-03-04T16:13:18.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e042adc2f3ba43-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.381 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:18.381 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.381 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.381 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.382 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.382 DEBUG:boto:Method: GET 2024-03-04T16:13:18.382 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.382 DEBUG:boto:Data: 2024-03-04T16:13:18.382 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.382 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.382 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.382 DEBUG:boto:Params: {} 2024-03-04T16:13:18.382 DEBUG:boto:Token: None 2024-03-04T16:13:18.383 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.383 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8= 2024-03-04T16:13:18.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8=', 'Content-Length': '0'} 2024-03-04T16:13:18.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000464a71ed69d2c5d9-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.385 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.385 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.385 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.385 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.385 DEBUG:boto:Method: GET 2024-03-04T16:13:18.385 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.386 DEBUG:boto:Data: 2024-03-04T16:13:18.386 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.386 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.386 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.386 DEBUG:boto:Params: {} 2024-03-04T16:13:18.386 DEBUG:boto:Token: None 2024-03-04T16:13:18.386 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.386 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8= 2024-03-04T16:13:18.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8=', 'Content-Length': '0'} 2024-03-04T16:13:18.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070cb220e0d26bdfc-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.388 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.388 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.388 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.389 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.389 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.389 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.389 DEBUG:boto:Data: 2024-03-04T16:13:18.389 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.389 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.389 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.389 DEBUG:boto:Params: {} 2024-03-04T16:13:18.389 DEBUG:boto:Token: None 2024-03-04T16:13:18.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.389 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AfAQ+vfgJb/HVnDuZH9WInDw48Q= 2024-03-04T16:13:18.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AfAQ+vfgJb/HVnDuZH9WInDw48Q=', 'Content-Length': '0'} 2024-03-04T16:13:18.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d0cb93b48791846-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.392 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.392 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.392 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.392 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.392 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.392 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.392 DEBUG:boto:Data: 2024-03-04T16:13:18.392 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.392 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.393 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.393 DEBUG:boto:Params: {} 2024-03-04T16:13:18.393 DEBUG:boto:Token: None 2024-03-04T16:13:18.393 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.393 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AfAQ+vfgJb/HVnDuZH9WInDw48Q= 2024-03-04T16:13:18.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AfAQ+vfgJb/HVnDuZH9WInDw48Q=', 'Content-Length': '0'} 2024-03-04T16:13:18.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccec6f0239297f43-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:18.396 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.396 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.396 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.396 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.396 DEBUG:boto:Method: GET 2024-03-04T16:13:18.396 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.396 DEBUG:boto:Data: 2024-03-04T16:13:18.396 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.396 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.396 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.397 DEBUG:boto:Params: {} 2024-03-04T16:13:18.397 DEBUG:boto:Token: None 2024-03-04T16:13:18.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8= 2024-03-04T16:13:18.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8=', 'Content-Length': '0'} 2024-03-04T16:13:18.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170c564b79b9ed9d-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.399 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.400 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.400 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.400 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.400 DEBUG:boto:Method: GET 2024-03-04T16:13:18.400 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.400 DEBUG:boto:Data: 2024-03-04T16:13:18.400 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.400 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.401 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.401 DEBUG:boto:Params: {} 2024-03-04T16:13:18.401 DEBUG:boto:Token: None 2024-03-04T16:13:18.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:18.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8= 2024-03-04T16:13:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PPEOdxvIVWw5NImCT2+/s3czza8=', 'Content-Length': '0'} 2024-03-04T16:13:18.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005da1c6e5d98c971f-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.403 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.403 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.403 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.403 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.403 DEBUG:boto:Method: GET 2024-03-04T16:13:18.403 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.403 DEBUG:boto:Data: 2024-03-04T16:13:18.403 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.404 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.404 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.404 DEBUG:boto:Params: {} 2024-03-04T16:13:18.404 DEBUG:boto:Token: None 2024-03-04T16:13:18.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.404 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b19e71736f4e04-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.407 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.407 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.407 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.407 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.407 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.408 DEBUG:boto:Method: GET 2024-03-04T16:13:18.408 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.408 DEBUG:boto:Data: 2024-03-04T16:13:18.408 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.408 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.408 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.408 DEBUG:boto:Params: {} 2024-03-04T16:13:18.408 DEBUG:boto:Token: None 2024-03-04T16:13:18.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.408 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ef5c0b8f4a9245a-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.411 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.411 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.411 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.411 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.412 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:18.412 DEBUG:boto:Data: 2024-03-04T16:13:18.412 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.412 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.412 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.412 DEBUG:boto:Params: {} 2024-03-04T16:13:18.412 DEBUG:boto:Token: None 2024-03-04T16:13:18.412 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj 2024-03-04T16:13:18.412 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3gmgQFOD8xa2VQI3Hv5XMnzxKY= 2024-03-04T16:13:18.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3gmgQFOD8xa2VQI3Hv5XMnzxKY=', 'Content-Length': '0'} 2024-03-04T16:13:18.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b5240407ca8f869-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.414 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.414 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.415 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.415 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:18.415 DEBUG:boto:Data: 2024-03-04T16:13:18.415 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.415 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.415 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.415 DEBUG:boto:Params: {} 2024-03-04T16:13:18.415 DEBUG:boto:Token: None 2024-03-04T16:13:18.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj 2024-03-04T16:13:18.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3gmgQFOD8xa2VQI3Hv5XMnzxKY= 2024-03-04T16:13:18.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3gmgQFOD8xa2VQI3Hv5XMnzxKY=', 'Content-Length': '0'} 2024-03-04T16:13:18.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d70ab3baa4409e-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:18.417 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.418 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.418 DEBUG:boto:Method: GET 2024-03-04T16:13:18.418 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:18.418 DEBUG:boto:Data: 2024-03-04T16:13:18.418 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.418 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.418 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.418 DEBUG:boto:Params: {} 2024-03-04T16:13:18.418 DEBUG:boto:Token: None 2024-03-04T16:13:18.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj 2024-03-04T16:13:18.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hUfDNblGk/7AQ82KEQV9jUCI1bY= 2024-03-04T16:13:18.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hUfDNblGk/7AQ82KEQV9jUCI1bY=', 'Content-Length': '0'} 2024-03-04T16:13:18.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0114bdb1ed3afae-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.421 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:18.421 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:18.421 DEBUG:boto:Method: GET 2024-03-04T16:13:18.421 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:18.421 DEBUG:boto:Data: 2024-03-04T16:13:18.421 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.421 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.421 DEBUG:boto:Params: {} 2024-03-04T16:13:18.422 DEBUG:boto:Token: None 2024-03-04T16:13:18.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/myobj 2024-03-04T16:13:18.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hUfDNblGk/7AQ82KEQV9jUCI1bY= 2024-03-04T16:13:18.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hUfDNblGk/7AQ82KEQV9jUCI1bY=', 'Content-Length': '0'} 2024-03-04T16:13:18.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061bde984ef65fa4c-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.424 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.424 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.424 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.424 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.424 DEBUG:boto:Method: GET 2024-03-04T16:13:18.424 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.424 DEBUG:boto:Data: 2024-03-04T16:13:18.424 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.424 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:13:18.425 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.425 DEBUG:boto:Params: {} 2024-03-04T16:13:18.425 DEBUG:boto:Token: None 2024-03-04T16:13:18.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002358b92e9e68fa7e-0065e5f31e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.468 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.469 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.469 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.469 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.469 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.469 DEBUG:boto:Method: GET 2024-03-04T16:13:18.469 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:18.469 DEBUG:boto:Data: 2024-03-04T16:13:18.469 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.470 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.470 DEBUG:boto:Params: {} 2024-03-04T16:13:18.470 DEBUG:boto:Token: None 2024-03-04T16:13:18.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b663b393bdbe8e0-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.472 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-10 zones={test-zone2, test-zone1} 2024-03-04T16:13:18.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:18.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:18.473 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:13:18.528 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:18.528 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:18.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:18.577+0000 7fc864610800 1 bucket sync caught up with source: 2024-03-04T16:13:18.579 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.18.6, , 00000000001.16.6, , 00000000001.15.6, 00000000002.14.6, 00000000001.16.6, 00000000001.12.6, 00000000001.20.6, ] 2024-03-04T16:13:18.579 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.18.6, , 00000000001.16.6, , 00000000001.15.6, 00000000002.14.6, 00000000001.16.6, 00000000001.12.6, 00000000001.20.6, ] 2024-03-04T16:13:18.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:18.577+0000 7fc864610800 0 bucket checkpoint complete 2024-03-04T16:13:18.595 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-10 zones={test-zone3, test-zone1} 2024-03-04T16:13:18.596 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.596 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.596 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.596 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:18.596 DEBUG:boto:Data: 2024-03-04T16:13:18.596 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.596 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.596 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.596 DEBUG:boto:Params: {} 2024-03-04T16:13:18.597 DEBUG:boto:Token: None 2024-03-04T16:13:18.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TLx/XBSYOQZcSuniUDFC2II3G6I= 2024-03-04T16:13:18.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TLx/XBSYOQZcSuniUDFC2II3G6I=', 'Content-Length': '0'} 2024-03-04T16:13:18.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000903b285376bec21f-0065e5f31e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.599 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.600 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.600 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.600 DEBUG:boto:Path: /sbnvju-10/ 2024-03-04T16:13:18.600 DEBUG:boto:Data: 2024-03-04T16:13:18.600 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.600 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.600 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.600 DEBUG:boto:Params: {} 2024-03-04T16:13:18.600 DEBUG:boto:Token: None 2024-03-04T16:13:18.600 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TLx/XBSYOQZcSuniUDFC2II3G6I= 2024-03-04T16:13:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TLx/XBSYOQZcSuniUDFC2II3G6I=', 'Content-Length': '0'} 2024-03-04T16:13:18.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000382223f0b243738a-0065e5f31e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.603 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:13:18.603 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.603 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.603 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.603 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.604 DEBUG:boto:Method: GET 2024-03-04T16:13:18.604 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.604 DEBUG:boto:Data: 2024-03-04T16:13:18.604 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.604 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.604 DEBUG:boto:Params: {} 2024-03-04T16:13:18.604 DEBUG:boto:Token: None 2024-03-04T16:13:18.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/?versions 2024-03-04T16:13:18.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA= 2024-03-04T16:13:18.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA=', 'Content-Length': '0'} 2024-03-04T16:13:18.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e776e39cc07801-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.607 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:13:18.608 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:13:18.608 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.609 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.609 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.609 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.609 DEBUG:boto:Method: GET 2024-03-04T16:13:18.609 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.609 DEBUG:boto:Data: 2024-03-04T16:13:18.609 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.609 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.609 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.609 DEBUG:boto:Params: {} 2024-03-04T16:13:18.609 DEBUG:boto:Token: None 2024-03-04T16:13:18.610 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/?versions 2024-03-04T16:13:18.610 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA= 2024-03-04T16:13:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA=', 'Content-Length': '0'} 2024-03-04T16:13:18.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051e6a7a2b475f713-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.612 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:13:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:13:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:13:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:13:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:13:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:13:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:13:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:13:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:13:18.614 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.614 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.614 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.614 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.614 DEBUG:boto:Method: GET 2024-03-04T16:13:18.614 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.614 DEBUG:boto:Data: 2024-03-04T16:13:18.615 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.615 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.615 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.615 DEBUG:boto:Params: {} 2024-03-04T16:13:18.615 DEBUG:boto:Token: None 2024-03-04T16:13:18.615 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/?versions 2024-03-04T16:13:18.615 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA= 2024-03-04T16:13:18.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA=', 'Content-Length': '0'} 2024-03-04T16:13:18.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd501b6b573352a9-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.618 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.619 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.619 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.619 DEBUG:boto:path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.619 DEBUG:boto:auth_path=/sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.619 DEBUG:boto:Method: GET 2024-03-04T16:13:18.619 DEBUG:boto:Path: /sbnvju-10/?versions&max-keys=999 2024-03-04T16:13:18.619 DEBUG:boto:Data: 2024-03-04T16:13:18.619 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.619 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.619 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.620 DEBUG:boto:Params: {} 2024-03-04T16:13:18.620 DEBUG:boto:Token: None 2024-03-04T16:13:18.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/?versions 2024-03-04T16:13:18.620 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA= 2024-03-04T16:13:18.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pJHxlXNnpwz/Gt2b9AHXaiPVYvA=', 'Content-Length': '0'} 2024-03-04T16:13:18.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013581c322de611f1-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.622 DEBUG:boto:b'sbnvju-10999false&nulltrue2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:18.623 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.623 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.623 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.624 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.624 DEBUG:boto:Method: GET 2024-03-04T16:13:18.624 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.624 DEBUG:boto:Data: 2024-03-04T16:13:18.624 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.624 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.624 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.624 DEBUG:boto:Params: {} 2024-03-04T16:13:18.624 DEBUG:boto:Token: None 2024-03-04T16:13:18.624 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ= 2024-03-04T16:13:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d46bf758a6ed4eb-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.627 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.627 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.627 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.627 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.627 DEBUG:boto:Method: GET 2024-03-04T16:13:18.627 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.627 DEBUG:boto:Data: 2024-03-04T16:13:18.628 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.628 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.628 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.628 DEBUG:boto:Params: {} 2024-03-04T16:13:18.628 DEBUG:boto:Token: None 2024-03-04T16:13:18.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.628 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ= 2024-03-04T16:13:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001af547a14e8eb67c-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.672 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.672 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.672 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.672 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.673 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.673 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.673 DEBUG:boto:Data: 2024-03-04T16:13:18.673 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.673 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.673 DEBUG:boto:Params: {} 2024-03-04T16:13:18.673 DEBUG:boto:Token: None 2024-03-04T16:13:18.673 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IIPwUp/dxxzlZhhAwkTXtUsEdog= 2024-03-04T16:13:18.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IIPwUp/dxxzlZhhAwkTXtUsEdog=', 'Content-Length': '0'} 2024-03-04T16:13:18.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003311fa37f63db3de-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.676 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.676 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.676 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.676 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.676 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.676 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.676 DEBUG:boto:Data: 2024-03-04T16:13:18.677 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.677 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.677 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.677 DEBUG:boto:Params: {} 2024-03-04T16:13:18.677 DEBUG:boto:Token: None 2024-03-04T16:13:18.677 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.677 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IIPwUp/dxxzlZhhAwkTXtUsEdog= 2024-03-04T16:13:18.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IIPwUp/dxxzlZhhAwkTXtUsEdog=', 'Content-Length': '0'} 2024-03-04T16:13:18.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ec5768110b8f78f-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:18.679 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.679 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.680 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.680 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.680 DEBUG:boto:Method: GET 2024-03-04T16:13:18.680 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.680 DEBUG:boto:Data: 2024-03-04T16:13:18.680 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.680 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.680 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.680 DEBUG:boto:Params: {} 2024-03-04T16:13:18.680 DEBUG:boto:Token: None 2024-03-04T16:13:18.681 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.681 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ= 2024-03-04T16:13:18.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c98d6e822271d6ce-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.728 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.728 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.728 DEBUG:boto:path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.728 DEBUG:boto:auth_path=/sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.729 DEBUG:boto:Method: GET 2024-03-04T16:13:18.729 DEBUG:boto:Path: /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.729 DEBUG:boto:Data: 2024-03-04T16:13:18.729 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.729 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.729 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.729 DEBUG:boto:Params: {} 2024-03-04T16:13:18.729 DEBUG:boto:Token: None 2024-03-04T16:13:18.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26?versionId=null 2024-03-04T16:13:18.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ= 2024-03-04T16:13:18.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8bdBxwJgVqEK9o+vNvGkwsMmaGQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dc326fc775ee7da-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.732 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.732 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.732 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.732 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.732 DEBUG:boto:Method: GET 2024-03-04T16:13:18.732 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:18.732 DEBUG:boto:Data: 2024-03-04T16:13:18.732 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.732 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.732 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.732 DEBUG:boto:Params: {} 2024-03-04T16:13:18.733 DEBUG:boto:Token: None 2024-03-04T16:13:18.733 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.733 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4567fe02474010a-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.735 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.736 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.736 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.736 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.736 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.736 DEBUG:boto:Method: GET 2024-03-04T16:13:18.736 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:18.736 DEBUG:boto:Data: 2024-03-04T16:13:18.736 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.736 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.736 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.736 DEBUG:boto:Params: {} 2024-03-04T16:13:18.737 DEBUG:boto:Token: None 2024-03-04T16:13:18.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089cc715b731d732-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.739 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.739 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.740 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.740 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.740 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:18.740 DEBUG:boto:Data: 2024-03-04T16:13:18.740 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.740 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.740 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.741 DEBUG:boto:Params: {} 2024-03-04T16:13:18.741 DEBUG:boto:Token: None 2024-03-04T16:13:18.741 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26 2024-03-04T16:13:18.741 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lwk8KYGH+M3iTfXMbm87XvtgW10= 2024-03-04T16:13:18.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lwk8KYGH+M3iTfXMbm87XvtgW10=', 'Content-Length': '0'} 2024-03-04T16:13:18.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1e2b543ce582d6-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.744 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.744 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.744 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.744 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:18.744 DEBUG:boto:Data: 2024-03-04T16:13:18.744 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.744 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.744 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.745 DEBUG:boto:Params: {} 2024-03-04T16:13:18.745 DEBUG:boto:Token: None 2024-03-04T16:13:18.745 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26 2024-03-04T16:13:18.745 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lwk8KYGH+M3iTfXMbm87XvtgW10= 2024-03-04T16:13:18.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lwk8KYGH+M3iTfXMbm87XvtgW10=', 'Content-Length': '0'} 2024-03-04T16:13:18.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003afa8f1ac3a8ee99-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:13:18.747 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.747 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.747 DEBUG:boto:Method: GET 2024-03-04T16:13:18.747 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:18.747 DEBUG:boto:Data: 2024-03-04T16:13:18.747 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.747 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.748 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.748 DEBUG:boto:Params: {} 2024-03-04T16:13:18.748 DEBUG:boto:Token: None 2024-03-04T16:13:18.748 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26 2024-03-04T16:13:18.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uIk6p1PkMXZpDtj+5A7CeOkCGs= 2024-03-04T16:13:18.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uIk6p1PkMXZpDtj+5A7CeOkCGs=', 'Content-Length': '0'} 2024-03-04T16:13:18.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003169c12a666b4bc4-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.792 DEBUG:boto:path=/sbnvju-10/%26 2024-03-04T16:13:18.792 DEBUG:boto:auth_path=/sbnvju-10/%26 2024-03-04T16:13:18.792 DEBUG:boto:Method: GET 2024-03-04T16:13:18.792 DEBUG:boto:Path: /sbnvju-10/%26 2024-03-04T16:13:18.793 DEBUG:boto:Data: 2024-03-04T16:13:18.793 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.793 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.793 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.793 DEBUG:boto:Params: {} 2024-03-04T16:13:18.793 DEBUG:boto:Token: None 2024-03-04T16:13:18.793 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/%26 2024-03-04T16:13:18.793 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uIk6p1PkMXZpDtj+5A7CeOkCGs= 2024-03-04T16:13:18.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uIk6p1PkMXZpDtj+5A7CeOkCGs=', 'Content-Length': '0'} 2024-03-04T16:13:18.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b120d1c0fef88b2-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.795 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.795 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.795 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.796 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.796 DEBUG:boto:Method: GET 2024-03-04T16:13:18.796 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:18.796 DEBUG:boto:Data: 2024-03-04T16:13:18.796 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.796 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.796 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.796 DEBUG:boto:Params: {} 2024-03-04T16:13:18.796 DEBUG:boto:Token: None 2024-03-04T16:13:18.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.797 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb125f559a1bb000-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.840 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.841 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.841 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.841 DEBUG:boto:path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.841 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%26 2024-03-04T16:13:18.841 DEBUG:boto:Method: GET 2024-03-04T16:13:18.841 DEBUG:boto:Path: /sbnvju-10/?prefix=%26 2024-03-04T16:13:18.841 DEBUG:boto:Data: 2024-03-04T16:13:18.841 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.841 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.842 DEBUG:boto:Params: {} 2024-03-04T16:13:18.842 DEBUG:boto:Token: None 2024-03-04T16:13:18.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d53192147e182298-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.844 DEBUG:boto:b'sbnvju-10&1000false&2024-03-04T16:13:01.198Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:18.845 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.845 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.845 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.845 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.845 DEBUG:boto:Method: GET 2024-03-04T16:13:18.845 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.846 DEBUG:boto:Data: 2024-03-04T16:13:18.846 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.846 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.846 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.846 DEBUG:boto:Params: {} 2024-03-04T16:13:18.846 DEBUG:boto:Token: None 2024-03-04T16:13:18.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y= 2024-03-04T16:13:18.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y=', 'Content-Length': '0'} 2024-03-04T16:13:18.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b439a5a3986f51d0-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.896 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.896 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.897 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.897 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.897 DEBUG:boto:Method: GET 2024-03-04T16:13:18.897 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.897 DEBUG:boto:Data: 2024-03-04T16:13:18.897 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.897 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.897 DEBUG:boto:Params: {} 2024-03-04T16:13:18.897 DEBUG:boto:Token: None 2024-03-04T16:13:18.898 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y= 2024-03-04T16:13:18.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y=', 'Content-Length': '0'} 2024-03-04T16:13:18.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a656a37e6ab7b5e0-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.900 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.900 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.900 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.901 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.901 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.901 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.901 DEBUG:boto:Data: 2024-03-04T16:13:18.901 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.901 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.901 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.901 DEBUG:boto:Params: {} 2024-03-04T16:13:18.901 DEBUG:boto:Token: None 2024-03-04T16:13:18.901 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.902 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yATa53H8209+M+crnvMBNYIwN8U= 2024-03-04T16:13:18.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yATa53H8209+M+crnvMBNYIwN8U=', 'Content-Length': '0'} 2024-03-04T16:13:18.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b93458d7d29b0199-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.904 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.904 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.904 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.904 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.904 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.904 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.904 DEBUG:boto:Data: 2024-03-04T16:13:18.904 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.904 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.905 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.905 DEBUG:boto:Params: {} 2024-03-04T16:13:18.905 DEBUG:boto:Token: None 2024-03-04T16:13:18.905 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.905 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yATa53H8209+M+crnvMBNYIwN8U= 2024-03-04T16:13:18.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yATa53H8209+M+crnvMBNYIwN8U=', 'Content-Length': '0'} 2024-03-04T16:13:18.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000045ba70528f61c90-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:18.907 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.907 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.907 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.907 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.907 DEBUG:boto:Method: GET 2024-03-04T16:13:18.907 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.907 DEBUG:boto:Data: 2024-03-04T16:13:18.907 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.907 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.907 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.908 DEBUG:boto:Params: {} 2024-03-04T16:13:18.908 DEBUG:boto:Token: None 2024-03-04T16:13:18.908 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.908 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y= 2024-03-04T16:13:18.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y=', 'Content-Length': '0'} 2024-03-04T16:13:18.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000903a87492427d595-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.910 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.910 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.910 DEBUG:boto:path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.910 DEBUG:boto:auth_path=/sbnvju-10/.?versionId=null 2024-03-04T16:13:18.910 DEBUG:boto:Method: GET 2024-03-04T16:13:18.910 DEBUG:boto:Path: /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.910 DEBUG:boto:Data: 2024-03-04T16:13:18.910 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.910 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.911 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.911 DEBUG:boto:Params: {} 2024-03-04T16:13:18.911 DEBUG:boto:Token: None 2024-03-04T16:13:18.911 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/.?versionId=null 2024-03-04T16:13:18.911 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y= 2024-03-04T16:13:18.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JRc0eHoya1afVefd+Z64CjPnt2Y=', 'Content-Length': '0'} 2024-03-04T16:13:18.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000871f349c86f3a0e1-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.913 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.913 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.913 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.914 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.914 DEBUG:boto:Method: GET 2024-03-04T16:13:18.914 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:18.914 DEBUG:boto:Data: 2024-03-04T16:13:18.914 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.914 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.914 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.914 DEBUG:boto:Params: {} 2024-03-04T16:13:18.914 DEBUG:boto:Token: None 2024-03-04T16:13:18.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.915 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd7844550fbef4e0-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.917 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.917 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.918 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.918 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.918 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.918 DEBUG:boto:Method: GET 2024-03-04T16:13:18.918 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:18.918 DEBUG:boto:Data: 2024-03-04T16:13:18.918 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.918 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.918 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.918 DEBUG:boto:Params: {} 2024-03-04T16:13:18.918 DEBUG:boto:Token: None 2024-03-04T16:13:18.919 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.919 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f71a98140c3377-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.921 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.922 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.922 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.922 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.922 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:18.922 DEBUG:boto:Data: 2024-03-04T16:13:18.922 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.922 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.923 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.923 DEBUG:boto:Params: {} 2024-03-04T16:13:18.923 DEBUG:boto:Token: None 2024-03-04T16:13:18.923 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/. 2024-03-04T16:13:18.923 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SfnclvB+P01jTwQeCX/y3DwmJwg= 2024-03-04T16:13:18.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SfnclvB+P01jTwQeCX/y3DwmJwg=', 'Content-Length': '0'} 2024-03-04T16:13:18.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a388c28d411918b0-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.925 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.925 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.925 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.925 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:18.925 DEBUG:boto:Data: 2024-03-04T16:13:18.926 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.926 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.926 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.926 DEBUG:boto:Params: {} 2024-03-04T16:13:18.926 DEBUG:boto:Token: None 2024-03-04T16:13:18.926 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/. 2024-03-04T16:13:18.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SfnclvB+P01jTwQeCX/y3DwmJwg= 2024-03-04T16:13:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SfnclvB+P01jTwQeCX/y3DwmJwg=', 'Content-Length': '0'} 2024-03-04T16:13:18.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c43bc4724ed078b9-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:13:18.928 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.928 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.928 DEBUG:boto:Method: GET 2024-03-04T16:13:18.928 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:18.929 DEBUG:boto:Data: 2024-03-04T16:13:18.929 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.929 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.929 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.929 DEBUG:boto:Params: {} 2024-03-04T16:13:18.929 DEBUG:boto:Token: None 2024-03-04T16:13:18.929 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/. 2024-03-04T16:13:18.929 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ivwxCUHn/EpIL69ZQwpXRHsJ9MQ= 2024-03-04T16:13:18.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ivwxCUHn/EpIL69ZQwpXRHsJ9MQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046e13c43bef3454f-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.972 DEBUG:boto:path=/sbnvju-10/. 2024-03-04T16:13:18.972 DEBUG:boto:auth_path=/sbnvju-10/. 2024-03-04T16:13:18.972 DEBUG:boto:Method: GET 2024-03-04T16:13:18.972 DEBUG:boto:Path: /sbnvju-10/. 2024-03-04T16:13:18.973 DEBUG:boto:Data: 2024-03-04T16:13:18.973 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.973 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.973 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.973 DEBUG:boto:Params: {} 2024-03-04T16:13:18.973 DEBUG:boto:Token: None 2024-03-04T16:13:18.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/. 2024-03-04T16:13:18.973 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ivwxCUHn/EpIL69ZQwpXRHsJ9MQ= 2024-03-04T16:13:18.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ivwxCUHn/EpIL69ZQwpXRHsJ9MQ=', 'Content-Length': '0'} 2024-03-04T16:13:18.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068e87138c52257ae-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.975 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.976 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.976 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.976 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.976 DEBUG:boto:Method: GET 2024-03-04T16:13:18.976 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:18.976 DEBUG:boto:Data: 2024-03-04T16:13:18.976 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.976 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.976 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.976 DEBUG:boto:Params: {} 2024-03-04T16:13:18.976 DEBUG:boto:Token: None 2024-03-04T16:13:18.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.977 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2bbd4c748305cb1-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.979 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.980 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:18.980 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:18.980 DEBUG:boto:path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.980 DEBUG:boto:auth_path=/sbnvju-10/?prefix=. 2024-03-04T16:13:18.980 DEBUG:boto:Method: GET 2024-03-04T16:13:18.980 DEBUG:boto:Path: /sbnvju-10/?prefix=. 2024-03-04T16:13:18.980 DEBUG:boto:Data: 2024-03-04T16:13:18.980 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.981 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.981 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.981 DEBUG:boto:Params: {} 2024-03-04T16:13:18.981 DEBUG:boto:Token: None 2024-03-04T16:13:18.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/ 2024-03-04T16:13:18.982 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s= 2024-03-04T16:13:18.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Ply0F7nOujXOCNPK3lY75jO49s=', 'Content-Length': '0'} 2024-03-04T16:13:18.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e20499f630211d14-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.984 DEBUG:boto:b'sbnvju-10.1000false.2024-03-04T16:13:01.210Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:18.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:18.985 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:18.985 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:18.985 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.985 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.985 DEBUG:boto:Method: GET 2024-03-04T16:13:18.985 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.985 DEBUG:boto:Data: 2024-03-04T16:13:18.985 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.985 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.986 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.986 DEBUG:boto:Params: {} 2024-03-04T16:13:18.986 DEBUG:boto:Token: None 2024-03-04T16:13:18.986 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.986 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nH3TLiZjHtjFsK+YF+quAjSaEzE= 2024-03-04T16:13:18.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nH3TLiZjHtjFsK+YF+quAjSaEzE=', 'Content-Length': '0'} 2024-03-04T16:13:18.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000740a18b2b5907c03-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.988 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:18.988 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:18.989 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.989 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.989 DEBUG:boto:Method: GET 2024-03-04T16:13:18.989 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.989 DEBUG:boto:Data: 2024-03-04T16:13:18.989 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.989 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.989 DEBUG:boto:Params: {} 2024-03-04T16:13:18.990 DEBUG:boto:Token: None 2024-03-04T16:13:18.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nH3TLiZjHtjFsK+YF+quAjSaEzE= 2024-03-04T16:13:18.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nH3TLiZjHtjFsK+YF+quAjSaEzE=', 'Content-Length': '0'} 2024-03-04T16:13:18.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5936882ab1aaa6e-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.992 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:18.992 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:18.993 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.993 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.993 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.993 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.994 DEBUG:boto:Data: 2024-03-04T16:13:18.994 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.994 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:18.994 DEBUG:boto:Port: 8001 2024-03-04T16:13:18.994 DEBUG:boto:Params: {} 2024-03-04T16:13:18.994 DEBUG:boto:Token: None 2024-03-04T16:13:18.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sbxlh8VAuPjobnzI8Y5gJgoGwCM= 2024-03-04T16:13:18.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sbxlh8VAuPjobnzI8Y5gJgoGwCM=', 'Content-Length': '0'} 2024-03-04T16:13:18.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077b2ae4788d6ba30-0065e5f31e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:18.996 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:18.997 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:18.997 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.997 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:18.997 DEBUG:boto:Method: HEAD 2024-03-04T16:13:18.997 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.997 DEBUG:boto:Data: 2024-03-04T16:13:18.997 DEBUG:boto:Headers: {} 2024-03-04T16:13:18.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:18.997 DEBUG:boto:Port: 8000 2024-03-04T16:13:18.997 DEBUG:boto:Params: {} 2024-03-04T16:13:18.997 DEBUG:boto:Token: None 2024-03-04T16:13:18.998 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:18 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:18.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sbxlh8VAuPjobnzI8Y5gJgoGwCM= 2024-03-04T16:13:18.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sbxlh8VAuPjobnzI8Y5gJgoGwCM=', 'Content-Length': '0'} 2024-03-04T16:13:18.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000135c6051ddfe2336-0065e5f31e-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:19.000 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.000 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.000 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:19.000 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:19.000 DEBUG:boto:Method: GET 2024-03-04T16:13:19.000 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:19.000 DEBUG:boto:Data: 2024-03-04T16:13:19.000 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.000 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.001 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.001 DEBUG:boto:Params: {} 2024-03-04T16:13:19.001 DEBUG:boto:Token: None 2024-03-04T16:13:19.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:19.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hFUO0AAQWq18o4/pjiFW3XElO3c= 2024-03-04T16:13:19.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hFUO0AAQWq18o4/pjiFW3XElO3c=', 'Content-Length': '0'} 2024-03-04T16:13:19.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d4fda3bd5c32fa7-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.048 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.048 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.048 DEBUG:boto:path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:19.049 DEBUG:boto:auth_path=/sbnvju-10/..?versionId=null 2024-03-04T16:13:19.049 DEBUG:boto:Method: GET 2024-03-04T16:13:19.049 DEBUG:boto:Path: /sbnvju-10/..?versionId=null 2024-03-04T16:13:19.049 DEBUG:boto:Data: 2024-03-04T16:13:19.049 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.049 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.049 DEBUG:boto:Params: {} 2024-03-04T16:13:19.049 DEBUG:boto:Token: None 2024-03-04T16:13:19.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/..?versionId=null 2024-03-04T16:13:19.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hFUO0AAQWq18o4/pjiFW3XElO3c= 2024-03-04T16:13:19.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hFUO0AAQWq18o4/pjiFW3XElO3c=', 'Content-Length': '0'} 2024-03-04T16:13:19.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e00f263285168e0-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.052 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.052 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.052 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.052 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.052 DEBUG:boto:Method: GET 2024-03-04T16:13:19.052 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:19.053 DEBUG:boto:Data: 2024-03-04T16:13:19.053 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.053 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.053 DEBUG:boto:Params: {} 2024-03-04T16:13:19.053 DEBUG:boto:Token: None 2024-03-04T16:13:19.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc9e7064196e6b3-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.056 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.057 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.057 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.057 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.057 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.058 DEBUG:boto:Method: GET 2024-03-04T16:13:19.058 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:19.058 DEBUG:boto:Data: 2024-03-04T16:13:19.058 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.058 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.058 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.058 DEBUG:boto:Params: {} 2024-03-04T16:13:19.058 DEBUG:boto:Token: None 2024-03-04T16:13:19.059 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.059 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2c1c5b4d451d99-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.062 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.063 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.063 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.063 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.063 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:19.063 DEBUG:boto:Data: 2024-03-04T16:13:19.063 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.063 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.063 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.063 DEBUG:boto:Params: {} 2024-03-04T16:13:19.063 DEBUG:boto:Token: None 2024-03-04T16:13:19.064 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.. 2024-03-04T16:13:19.064 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HtVPrYmIwo6WjsiMNkxQI+ZdH8Y= 2024-03-04T16:13:19.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HtVPrYmIwo6WjsiMNkxQI+ZdH8Y=', 'Content-Length': '0'} 2024-03-04T16:13:19.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e7c85d9087f6b7b-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.066 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.066 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.066 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.066 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:19.066 DEBUG:boto:Data: 2024-03-04T16:13:19.066 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.066 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.066 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.066 DEBUG:boto:Params: {} 2024-03-04T16:13:19.066 DEBUG:boto:Token: None 2024-03-04T16:13:19.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.. 2024-03-04T16:13:19.067 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HtVPrYmIwo6WjsiMNkxQI+ZdH8Y= 2024-03-04T16:13:19.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HtVPrYmIwo6WjsiMNkxQI+ZdH8Y=', 'Content-Length': '0'} 2024-03-04T16:13:19.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e98260b7374abeb-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:13:19.068 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.068 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.069 DEBUG:boto:Method: GET 2024-03-04T16:13:19.069 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:19.069 DEBUG:boto:Data: 2024-03-04T16:13:19.069 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.069 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.069 DEBUG:boto:Params: {} 2024-03-04T16:13:19.069 DEBUG:boto:Token: None 2024-03-04T16:13:19.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.. 2024-03-04T16:13:19.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0+eaF3FfFYeZS9MBauPzHXdjqqQ= 2024-03-04T16:13:19.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0+eaF3FfFYeZS9MBauPzHXdjqqQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c11ca3937a14628-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.071 DEBUG:boto:path=/sbnvju-10/.. 2024-03-04T16:13:19.071 DEBUG:boto:auth_path=/sbnvju-10/.. 2024-03-04T16:13:19.072 DEBUG:boto:Method: GET 2024-03-04T16:13:19.072 DEBUG:boto:Path: /sbnvju-10/.. 2024-03-04T16:13:19.072 DEBUG:boto:Data: 2024-03-04T16:13:19.072 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.072 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.072 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.072 DEBUG:boto:Params: {} 2024-03-04T16:13:19.072 DEBUG:boto:Token: None 2024-03-04T16:13:19.072 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.. 2024-03-04T16:13:19.072 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0+eaF3FfFYeZS9MBauPzHXdjqqQ= 2024-03-04T16:13:19.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0+eaF3FfFYeZS9MBauPzHXdjqqQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f677a4d5f7f2f6ce-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.124 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.124 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.124 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.125 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.125 DEBUG:boto:Method: GET 2024-03-04T16:13:19.125 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:19.125 DEBUG:boto:Data: 2024-03-04T16:13:19.125 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.125 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.125 DEBUG:boto:Params: {} 2024-03-04T16:13:19.125 DEBUG:boto:Token: None 2024-03-04T16:13:19.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73640fc18f37c24-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.128 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.129 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.129 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.129 DEBUG:boto:path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.129 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.. 2024-03-04T16:13:19.129 DEBUG:boto:Method: GET 2024-03-04T16:13:19.129 DEBUG:boto:Path: /sbnvju-10/?prefix=.. 2024-03-04T16:13:19.129 DEBUG:boto:Data: 2024-03-04T16:13:19.129 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.129 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.129 DEBUG:boto:Params: {} 2024-03-04T16:13:19.130 DEBUG:boto:Token: None 2024-03-04T16:13:19.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb15573d5fd3738c-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.132 DEBUG:boto:b'sbnvju-10..1000false..2024-03-04T16:13:01.224Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:19.133 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.133 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.133 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.133 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.133 DEBUG:boto:Method: GET 2024-03-04T16:13:19.133 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.134 DEBUG:boto:Data: 2024-03-04T16:13:19.134 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.134 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.134 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.134 DEBUG:boto:Params: {} 2024-03-04T16:13:19.134 DEBUG:boto:Token: None 2024-03-04T16:13:19.134 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.134 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0= 2024-03-04T16:13:19.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0=', 'Content-Length': '0'} 2024-03-04T16:13:19.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096af629fce2cc85b-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.180 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.180 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.180 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.181 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.181 DEBUG:boto:Method: GET 2024-03-04T16:13:19.181 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.181 DEBUG:boto:Data: 2024-03-04T16:13:19.181 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.181 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.181 DEBUG:boto:Params: {} 2024-03-04T16:13:19.181 DEBUG:boto:Token: None 2024-03-04T16:13:19.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0= 2024-03-04T16:13:19.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0=', 'Content-Length': '0'} 2024-03-04T16:13:19.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024a88ac8d7bd605a-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.184 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.184 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.184 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.184 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.184 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.184 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.184 DEBUG:boto:Data: 2024-03-04T16:13:19.185 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.185 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.185 DEBUG:boto:Params: {} 2024-03-04T16:13:19.185 DEBUG:boto:Token: None 2024-03-04T16:13:19.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.185 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1RWuDtx81yCVpBkEkw8l1PpkCGQ= 2024-03-04T16:13:19.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1RWuDtx81yCVpBkEkw8l1PpkCGQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d7d6fcf567a3270-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.187 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.187 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.187 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.187 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.187 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.187 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.188 DEBUG:boto:Data: 2024-03-04T16:13:19.188 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.188 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.188 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.188 DEBUG:boto:Params: {} 2024-03-04T16:13:19.188 DEBUG:boto:Token: None 2024-03-04T16:13:19.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.188 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1RWuDtx81yCVpBkEkw8l1PpkCGQ= 2024-03-04T16:13:19.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1RWuDtx81yCVpBkEkw8l1PpkCGQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e22bb4f9be58cb1-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:19.190 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.190 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.190 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.190 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.190 DEBUG:boto:Method: GET 2024-03-04T16:13:19.191 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.191 DEBUG:boto:Data: 2024-03-04T16:13:19.191 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.191 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.191 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.191 DEBUG:boto:Params: {} 2024-03-04T16:13:19.191 DEBUG:boto:Token: None 2024-03-04T16:13:19.192 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.192 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0= 2024-03-04T16:13:19.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0=', 'Content-Length': '0'} 2024-03-04T16:13:19.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083eb95792f9872d5-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.195 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.196 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.196 DEBUG:boto:path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.196 DEBUG:boto:auth_path=/sbnvju-10/...?versionId=null 2024-03-04T16:13:19.196 DEBUG:boto:Method: GET 2024-03-04T16:13:19.196 DEBUG:boto:Path: /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.197 DEBUG:boto:Data: 2024-03-04T16:13:19.197 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.197 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.197 DEBUG:boto:Params: {} 2024-03-04T16:13:19.198 DEBUG:boto:Token: None 2024-03-04T16:13:19.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/...?versionId=null 2024-03-04T16:13:19.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0= 2024-03-04T16:13:19.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qHEt6O/6hDOkXPJogFaWqlm6+h0=', 'Content-Length': '0'} 2024-03-04T16:13:19.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088290dc0dde9d259-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.201 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.201 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.202 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.202 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.202 DEBUG:boto:Method: GET 2024-03-04T16:13:19.202 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:19.203 DEBUG:boto:Data: 2024-03-04T16:13:19.203 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.203 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.203 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.203 DEBUG:boto:Params: {} 2024-03-04T16:13:19.204 DEBUG:boto:Token: None 2024-03-04T16:13:19.204 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.204 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcdcfff5af1504e-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.207 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.208 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.208 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.208 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.208 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.209 DEBUG:boto:Method: GET 2024-03-04T16:13:19.209 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:19.209 DEBUG:boto:Data: 2024-03-04T16:13:19.209 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.209 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.210 DEBUG:boto:Params: {} 2024-03-04T16:13:19.210 DEBUG:boto:Token: None 2024-03-04T16:13:19.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e8ea5a5943de16-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.213 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.214 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.214 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.214 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.215 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:19.215 DEBUG:boto:Data: 2024-03-04T16:13:19.215 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.215 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.215 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.216 DEBUG:boto:Params: {} 2024-03-04T16:13:19.216 DEBUG:boto:Token: None 2024-03-04T16:13:19.216 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/... 2024-03-04T16:13:19.216 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OHbunOmmBKUj/2SlwJz6FzpnKrw= 2024-03-04T16:13:19.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OHbunOmmBKUj/2SlwJz6FzpnKrw=', 'Content-Length': '0'} 2024-03-04T16:13:19.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e41a2823d21d80de-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.219 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.219 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.219 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.219 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:19.219 DEBUG:boto:Data: 2024-03-04T16:13:19.219 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.219 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.220 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.220 DEBUG:boto:Params: {} 2024-03-04T16:13:19.220 DEBUG:boto:Token: None 2024-03-04T16:13:19.220 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/... 2024-03-04T16:13:19.221 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OHbunOmmBKUj/2SlwJz6FzpnKrw= 2024-03-04T16:13:19.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OHbunOmmBKUj/2SlwJz6FzpnKrw=', 'Content-Length': '0'} 2024-03-04T16:13:19.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006854b3e000895202-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:13:19.223 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.223 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.223 DEBUG:boto:Method: GET 2024-03-04T16:13:19.223 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:19.223 DEBUG:boto:Data: 2024-03-04T16:13:19.224 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.224 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.224 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.224 DEBUG:boto:Params: {} 2024-03-04T16:13:19.224 DEBUG:boto:Token: None 2024-03-04T16:13:19.224 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/... 2024-03-04T16:13:19.224 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RaQ1ZSQzGLPNU0ZKsR1AcmHfDD0= 2024-03-04T16:13:19.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RaQ1ZSQzGLPNU0ZKsR1AcmHfDD0=', 'Content-Length': '0'} 2024-03-04T16:13:19.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003a7ce706d2541c5-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.226 DEBUG:boto:path=/sbnvju-10/... 2024-03-04T16:13:19.226 DEBUG:boto:auth_path=/sbnvju-10/... 2024-03-04T16:13:19.226 DEBUG:boto:Method: GET 2024-03-04T16:13:19.227 DEBUG:boto:Path: /sbnvju-10/... 2024-03-04T16:13:19.227 DEBUG:boto:Data: 2024-03-04T16:13:19.227 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.227 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.227 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.227 DEBUG:boto:Params: {} 2024-03-04T16:13:19.227 DEBUG:boto:Token: None 2024-03-04T16:13:19.228 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/... 2024-03-04T16:13:19.228 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RaQ1ZSQzGLPNU0ZKsR1AcmHfDD0= 2024-03-04T16:13:19.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RaQ1ZSQzGLPNU0ZKsR1AcmHfDD0=', 'Content-Length': '0'} 2024-03-04T16:13:19.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5f18ca8d2d03dbb-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.230 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.230 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.230 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.230 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.230 DEBUG:boto:Method: GET 2024-03-04T16:13:19.231 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:19.231 DEBUG:boto:Data: 2024-03-04T16:13:19.231 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.231 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.231 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.231 DEBUG:boto:Params: {} 2024-03-04T16:13:19.231 DEBUG:boto:Token: None 2024-03-04T16:13:19.231 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.231 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a4f29be1d40f664-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.234 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.234 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.235 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.235 DEBUG:boto:path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.235 DEBUG:boto:auth_path=/sbnvju-10/?prefix=... 2024-03-04T16:13:19.235 DEBUG:boto:Method: GET 2024-03-04T16:13:19.235 DEBUG:boto:Path: /sbnvju-10/?prefix=... 2024-03-04T16:13:19.235 DEBUG:boto:Data: 2024-03-04T16:13:19.235 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.235 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.235 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.235 DEBUG:boto:Params: {} 2024-03-04T16:13:19.235 DEBUG:boto:Token: None 2024-03-04T16:13:19.236 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.236 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000484988e706511884-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.238 DEBUG:boto:b'sbnvju-10...1000false...2024-03-04T16:13:01.272Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:19.239 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.239 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.239 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.239 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.239 DEBUG:boto:Method: GET 2024-03-04T16:13:19.239 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.239 DEBUG:boto:Data: 2024-03-04T16:13:19.239 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.240 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.240 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.240 DEBUG:boto:Params: {} 2024-03-04T16:13:19.240 DEBUG:boto:Token: None 2024-03-04T16:13:19.240 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.240 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8= 2024-03-04T16:13:19.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8=', 'Content-Length': '0'} 2024-03-04T16:13:19.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ab8e66537599b6f-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.243 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.243 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.243 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.243 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.243 DEBUG:boto:Method: GET 2024-03-04T16:13:19.243 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.244 DEBUG:boto:Data: 2024-03-04T16:13:19.245 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.245 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.245 DEBUG:boto:Params: {} 2024-03-04T16:13:19.245 DEBUG:boto:Token: None 2024-03-04T16:13:19.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8= 2024-03-04T16:13:19.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8=', 'Content-Length': '0'} 2024-03-04T16:13:19.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096534862921fb9b5-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.248 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.248 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.248 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.248 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.248 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.248 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.249 DEBUG:boto:Data: 2024-03-04T16:13:19.249 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.249 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.249 DEBUG:boto:Params: {} 2024-03-04T16:13:19.249 DEBUG:boto:Token: None 2024-03-04T16:13:19.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cCZgNpPZb/x+vC5lsGWt0an6h5M= 2024-03-04T16:13:19.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cCZgNpPZb/x+vC5lsGWt0an6h5M=', 'Content-Length': '0'} 2024-03-04T16:13:19.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b43cb8c92c426ea-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.251 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.252 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.252 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.252 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.252 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.252 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.253 DEBUG:boto:Data: 2024-03-04T16:13:19.253 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.253 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.253 DEBUG:boto:Params: {} 2024-03-04T16:13:19.253 DEBUG:boto:Token: None 2024-03-04T16:13:19.254 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cCZgNpPZb/x+vC5lsGWt0an6h5M= 2024-03-04T16:13:19.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cCZgNpPZb/x+vC5lsGWt0an6h5M=', 'Content-Length': '0'} 2024-03-04T16:13:19.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc09376e920ba7bf-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:19.256 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.256 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.256 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.256 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.256 DEBUG:boto:Method: GET 2024-03-04T16:13:19.256 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.256 DEBUG:boto:Data: 2024-03-04T16:13:19.256 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.256 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.256 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.256 DEBUG:boto:Params: {} 2024-03-04T16:13:19.257 DEBUG:boto:Token: None 2024-03-04T16:13:19.257 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8= 2024-03-04T16:13:19.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8=', 'Content-Length': '0'} 2024-03-04T16:13:19.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000411919b2a96ada0-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.259 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.259 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.259 DEBUG:boto:path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.259 DEBUG:boto:auth_path=/sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.259 DEBUG:boto:Method: GET 2024-03-04T16:13:19.259 DEBUG:boto:Path: /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.259 DEBUG:boto:Data: 2024-03-04T16:13:19.260 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.260 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.260 DEBUG:boto:Params: {} 2024-03-04T16:13:19.260 DEBUG:boto:Token: None 2024-03-04T16:13:19.260 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o?versionId=null 2024-03-04T16:13:19.260 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8= 2024-03-04T16:13:19.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PGiV42TRT2YYLM96UuihDNNRzh8=', 'Content-Length': '0'} 2024-03-04T16:13:19.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8342b8b515f70e6-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.262 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.262 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.262 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.262 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.262 DEBUG:boto:Method: GET 2024-03-04T16:13:19.263 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:19.263 DEBUG:boto:Data: 2024-03-04T16:13:19.263 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.263 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.263 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.263 DEBUG:boto:Params: {} 2024-03-04T16:13:19.263 DEBUG:boto:Token: None 2024-03-04T16:13:19.263 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a4f2b773e4759b-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.266 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.267 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.267 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.267 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.267 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.267 DEBUG:boto:Method: GET 2024-03-04T16:13:19.267 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:19.267 DEBUG:boto:Data: 2024-03-04T16:13:19.267 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.267 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.267 DEBUG:boto:Params: {} 2024-03-04T16:13:19.267 DEBUG:boto:Token: None 2024-03-04T16:13:19.268 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.268 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007122b58376ec7ecd-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.270 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.272 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.272 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.272 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.272 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:19.272 DEBUG:boto:Data: 2024-03-04T16:13:19.272 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.272 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.272 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.273 DEBUG:boto:Params: {} 2024-03-04T16:13:19.273 DEBUG:boto:Token: None 2024-03-04T16:13:19.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o 2024-03-04T16:13:19.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JpmoJX6NTWYjMBCjEmPH2m1zgcA= 2024-03-04T16:13:19.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JpmoJX6NTWYjMBCjEmPH2m1zgcA=', 'Content-Length': '0'} 2024-03-04T16:13:19.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0ed4c6fb2b87d1c-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.275 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.275 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.276 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.276 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:19.276 DEBUG:boto:Data: 2024-03-04T16:13:19.276 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.276 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.276 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.276 DEBUG:boto:Params: {} 2024-03-04T16:13:19.276 DEBUG:boto:Token: None 2024-03-04T16:13:19.276 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o 2024-03-04T16:13:19.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JpmoJX6NTWYjMBCjEmPH2m1zgcA= 2024-03-04T16:13:19.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JpmoJX6NTWYjMBCjEmPH2m1zgcA=', 'Content-Length': '0'} 2024-03-04T16:13:19.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb925c351d81dc26-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:13:19.280 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.281 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.281 DEBUG:boto:Method: GET 2024-03-04T16:13:19.281 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:19.281 DEBUG:boto:Data: 2024-03-04T16:13:19.281 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.281 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.281 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.281 DEBUG:boto:Params: {} 2024-03-04T16:13:19.281 DEBUG:boto:Token: None 2024-03-04T16:13:19.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o 2024-03-04T16:13:19.282 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yp7/FgVbl9g7EE23plbWXWCLqvg= 2024-03-04T16:13:19.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yp7/FgVbl9g7EE23plbWXWCLqvg=', 'Content-Length': '0'} 2024-03-04T16:13:19.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002add7070b48cc148-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.283 DEBUG:boto:path=/sbnvju-10/.o 2024-03-04T16:13:19.284 DEBUG:boto:auth_path=/sbnvju-10/.o 2024-03-04T16:13:19.284 DEBUG:boto:Method: GET 2024-03-04T16:13:19.284 DEBUG:boto:Path: /sbnvju-10/.o 2024-03-04T16:13:19.284 DEBUG:boto:Data: 2024-03-04T16:13:19.284 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.284 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.284 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.284 DEBUG:boto:Params: {} 2024-03-04T16:13:19.284 DEBUG:boto:Token: None 2024-03-04T16:13:19.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o 2024-03-04T16:13:19.285 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yp7/FgVbl9g7EE23plbWXWCLqvg= 2024-03-04T16:13:19.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yp7/FgVbl9g7EE23plbWXWCLqvg=', 'Content-Length': '0'} 2024-03-04T16:13:19.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059bcfbb6257088e6-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.328 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.328 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.328 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.328 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.328 DEBUG:boto:Method: GET 2024-03-04T16:13:19.329 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:19.329 DEBUG:boto:Data: 2024-03-04T16:13:19.329 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.329 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.329 DEBUG:boto:Params: {} 2024-03-04T16:13:19.329 DEBUG:boto:Token: None 2024-03-04T16:13:19.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039af232e4ba1b568-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.333 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.334 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.334 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.334 DEBUG:boto:path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.334 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o 2024-03-04T16:13:19.334 DEBUG:boto:Method: GET 2024-03-04T16:13:19.334 DEBUG:boto:Path: /sbnvju-10/?prefix=.o 2024-03-04T16:13:19.334 DEBUG:boto:Data: 2024-03-04T16:13:19.334 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.334 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.334 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.335 DEBUG:boto:Params: {} 2024-03-04T16:13:19.335 DEBUG:boto:Token: None 2024-03-04T16:13:19.335 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.335 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000957846b4c0f5742e-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.337 DEBUG:boto:b'sbnvju-10.o1000false.o2024-03-04T16:13:01.281Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:19.338 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.338 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.338 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.338 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.338 DEBUG:boto:Method: GET 2024-03-04T16:13:19.338 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.338 DEBUG:boto:Data: 2024-03-04T16:13:19.338 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.339 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.339 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.339 DEBUG:boto:Params: {} 2024-03-04T16:13:19.339 DEBUG:boto:Token: None 2024-03-04T16:13:19.339 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.339 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo= 2024-03-04T16:13:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo=', 'Content-Length': '0'} 2024-03-04T16:13:19.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b445fd5fece623da-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.341 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.341 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.341 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.342 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.342 DEBUG:boto:Method: GET 2024-03-04T16:13:19.342 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.342 DEBUG:boto:Data: 2024-03-04T16:13:19.342 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.342 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.342 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.342 DEBUG:boto:Params: {} 2024-03-04T16:13:19.342 DEBUG:boto:Token: None 2024-03-04T16:13:19.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo= 2024-03-04T16:13:19.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo=', 'Content-Length': '0'} 2024-03-04T16:13:19.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a52e8a3a90329b08-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.344 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.344 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.345 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.345 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.345 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.345 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.345 DEBUG:boto:Data: 2024-03-04T16:13:19.345 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.345 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.345 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.345 DEBUG:boto:Params: {} 2024-03-04T16:13:19.345 DEBUG:boto:Token: None 2024-03-04T16:13:19.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.346 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FEs/jSXuXdJP2ePlwj2/UKvYjf4= 2024-03-04T16:13:19.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FEs/jSXuXdJP2ePlwj2/UKvYjf4=', 'Content-Length': '0'} 2024-03-04T16:13:19.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077c67ebcb23d1cbc-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.347 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.347 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.348 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.348 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.348 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.348 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.348 DEBUG:boto:Data: 2024-03-04T16:13:19.348 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.348 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.348 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.348 DEBUG:boto:Params: {} 2024-03-04T16:13:19.348 DEBUG:boto:Token: None 2024-03-04T16:13:19.349 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FEs/jSXuXdJP2ePlwj2/UKvYjf4= 2024-03-04T16:13:19.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FEs/jSXuXdJP2ePlwj2/UKvYjf4=', 'Content-Length': '0'} 2024-03-04T16:13:19.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052ff5274a500cee7-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:19.351 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.351 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.351 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.352 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.352 DEBUG:boto:Method: GET 2024-03-04T16:13:19.352 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.352 DEBUG:boto:Data: 2024-03-04T16:13:19.352 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.352 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.352 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.352 DEBUG:boto:Params: {} 2024-03-04T16:13:19.352 DEBUG:boto:Token: None 2024-03-04T16:13:19.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo= 2024-03-04T16:13:19.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo=', 'Content-Length': '0'} 2024-03-04T16:13:19.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1d19ee1eae8fb09-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.396 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.396 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.396 DEBUG:boto:path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.397 DEBUG:boto:auth_path=/sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.397 DEBUG:boto:Method: GET 2024-03-04T16:13:19.397 DEBUG:boto:Path: /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.397 DEBUG:boto:Data: 2024-03-04T16:13:19.397 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.397 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.397 DEBUG:boto:Params: {} 2024-03-04T16:13:19.397 DEBUG:boto:Token: None 2024-03-04T16:13:19.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o.?versionId=null 2024-03-04T16:13:19.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo= 2024-03-04T16:13:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cf9Fir2EdNa1OUuQjC6bM5knwEo=', 'Content-Length': '0'} 2024-03-04T16:13:19.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8af6bb186d4b69e-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.400 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.400 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.400 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.400 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.400 DEBUG:boto:Method: GET 2024-03-04T16:13:19.400 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.400 DEBUG:boto:Data: 2024-03-04T16:13:19.400 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.400 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.401 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.401 DEBUG:boto:Params: {} 2024-03-04T16:13:19.401 DEBUG:boto:Token: None 2024-03-04T16:13:19.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000839f11bb72106540-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.404 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.404 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.404 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.405 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.405 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.405 DEBUG:boto:Method: GET 2024-03-04T16:13:19.405 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.405 DEBUG:boto:Data: 2024-03-04T16:13:19.406 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.406 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.406 DEBUG:boto:Params: {} 2024-03-04T16:13:19.406 DEBUG:boto:Token: None 2024-03-04T16:13:19.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008529ebff71e44a7d-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.409 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.410 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.410 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.410 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.410 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:19.410 DEBUG:boto:Data: 2024-03-04T16:13:19.410 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.410 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.410 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.410 DEBUG:boto:Params: {} 2024-03-04T16:13:19.410 DEBUG:boto:Token: None 2024-03-04T16:13:19.411 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o. 2024-03-04T16:13:19.411 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXYMXEw3yp493lN3NC0Z0CajfoM= 2024-03-04T16:13:19.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXYMXEw3yp493lN3NC0Z0CajfoM=', 'Content-Length': '0'} 2024-03-04T16:13:19.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000312d4ac8646042a-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.413 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.413 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.414 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.414 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:19.414 DEBUG:boto:Data: 2024-03-04T16:13:19.414 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.414 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.414 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.414 DEBUG:boto:Params: {} 2024-03-04T16:13:19.414 DEBUG:boto:Token: None 2024-03-04T16:13:19.414 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o. 2024-03-04T16:13:19.414 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXYMXEw3yp493lN3NC0Z0CajfoM= 2024-03-04T16:13:19.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXYMXEw3yp493lN3NC0Z0CajfoM=', 'Content-Length': '0'} 2024-03-04T16:13:19.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000949d29b961ca601a-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:13:19.416 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.417 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.417 DEBUG:boto:Method: GET 2024-03-04T16:13:19.417 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:19.417 DEBUG:boto:Data: 2024-03-04T16:13:19.417 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.417 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.417 DEBUG:boto:Params: {} 2024-03-04T16:13:19.417 DEBUG:boto:Token: None 2024-03-04T16:13:19.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o. 2024-03-04T16:13:19.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0avv3GMg9KQbX4jT+rnrYsw7q/o= 2024-03-04T16:13:19.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0avv3GMg9KQbX4jT+rnrYsw7q/o=', 'Content-Length': '0'} 2024-03-04T16:13:19.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015a904a04039cfc3-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.420 DEBUG:boto:path=/sbnvju-10/.o. 2024-03-04T16:13:19.420 DEBUG:boto:auth_path=/sbnvju-10/.o. 2024-03-04T16:13:19.420 DEBUG:boto:Method: GET 2024-03-04T16:13:19.420 DEBUG:boto:Path: /sbnvju-10/.o. 2024-03-04T16:13:19.420 DEBUG:boto:Data: 2024-03-04T16:13:19.420 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.420 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.420 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.420 DEBUG:boto:Params: {} 2024-03-04T16:13:19.421 DEBUG:boto:Token: None 2024-03-04T16:13:19.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/.o. 2024-03-04T16:13:19.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0avv3GMg9KQbX4jT+rnrYsw7q/o= 2024-03-04T16:13:19.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0avv3GMg9KQbX4jT+rnrYsw7q/o=', 'Content-Length': '0'} 2024-03-04T16:13:19.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd994cd4c0a50ad5-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.424 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.424 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.424 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.424 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.424 DEBUG:boto:Method: GET 2024-03-04T16:13:19.424 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.424 DEBUG:boto:Data: 2024-03-04T16:13:19.424 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.424 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.424 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.424 DEBUG:boto:Params: {} 2024-03-04T16:13:19.425 DEBUG:boto:Token: None 2024-03-04T16:13:19.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064f578bccbbf0bc9-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.427 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.428 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.428 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.428 DEBUG:boto:path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.428 DEBUG:boto:auth_path=/sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.428 DEBUG:boto:Method: GET 2024-03-04T16:13:19.428 DEBUG:boto:Path: /sbnvju-10/?prefix=.o. 2024-03-04T16:13:19.428 DEBUG:boto:Data: 2024-03-04T16:13:19.428 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.428 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.428 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.428 DEBUG:boto:Params: {} 2024-03-04T16:13:19.429 DEBUG:boto:Token: None 2024-03-04T16:13:19.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e11b5c5e1238fc9-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.431 DEBUG:boto:b'sbnvju-10.o.1000false.o.2024-03-04T16:13:01.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:19.432 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.432 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.432 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.432 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.432 DEBUG:boto:Method: GET 2024-03-04T16:13:19.432 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.433 DEBUG:boto:Data: 2024-03-04T16:13:19.433 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.433 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.433 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.433 DEBUG:boto:Params: {} 2024-03-04T16:13:19.433 DEBUG:boto:Token: None 2024-03-04T16:13:19.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM= 2024-03-04T16:13:19.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM=', 'Content-Length': '0'} 2024-03-04T16:13:19.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f0b3ce8996ae5c9-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.436 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.436 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.436 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.436 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.436 DEBUG:boto:Method: GET 2024-03-04T16:13:19.436 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.436 DEBUG:boto:Data: 2024-03-04T16:13:19.436 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.436 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.436 DEBUG:boto:Params: {} 2024-03-04T16:13:19.437 DEBUG:boto:Token: None 2024-03-04T16:13:19.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM= 2024-03-04T16:13:19.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM=', 'Content-Length': '0'} 2024-03-04T16:13:19.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e0c4b7d9a30c3e6-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.439 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.439 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.439 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.439 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.439 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.439 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.439 DEBUG:boto:Data: 2024-03-04T16:13:19.439 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.439 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.439 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.439 DEBUG:boto:Params: {} 2024-03-04T16:13:19.439 DEBUG:boto:Token: None 2024-03-04T16:13:19.440 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.440 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s5UOf4i6q3nZHA8MikdF9oKzK4A= 2024-03-04T16:13:19.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s5UOf4i6q3nZHA8MikdF9oKzK4A=', 'Content-Length': '0'} 2024-03-04T16:13:19.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b1a4c6a6ab545cb-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.441 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.442 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.442 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.442 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.442 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.442 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.442 DEBUG:boto:Data: 2024-03-04T16:13:19.442 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.443 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.443 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.443 DEBUG:boto:Params: {} 2024-03-04T16:13:19.443 DEBUG:boto:Token: None 2024-03-04T16:13:19.443 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s5UOf4i6q3nZHA8MikdF9oKzK4A= 2024-03-04T16:13:19.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s5UOf4i6q3nZHA8MikdF9oKzK4A=', 'Content-Length': '0'} 2024-03-04T16:13:19.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216d8f30b9f1db1a-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:19.445 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.445 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.445 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.445 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.445 DEBUG:boto:Method: GET 2024-03-04T16:13:19.445 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.445 DEBUG:boto:Data: 2024-03-04T16:13:19.445 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.445 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.446 DEBUG:boto:Params: {} 2024-03-04T16:13:19.446 DEBUG:boto:Token: None 2024-03-04T16:13:19.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM= 2024-03-04T16:13:19.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM=', 'Content-Length': '0'} 2024-03-04T16:13:19.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087598528ef06cf3a-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.448 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.448 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.448 DEBUG:boto:path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.448 DEBUG:boto:auth_path=/sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.448 DEBUG:boto:Method: GET 2024-03-04T16:13:19.448 DEBUG:boto:Path: /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.448 DEBUG:boto:Data: 2024-03-04T16:13:19.448 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.448 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.448 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.448 DEBUG:boto:Params: {} 2024-03-04T16:13:19.449 DEBUG:boto:Token: None 2024-03-04T16:13:19.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A?versionId=null 2024-03-04T16:13:19.449 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM= 2024-03-04T16:13:19.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M8B2Gbi95ZcLZyhzAmqUUmuTKvM=', 'Content-Length': '0'} 2024-03-04T16:13:19.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052ef8b9e0db19156-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.450 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.450 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.451 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.451 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.451 DEBUG:boto:Method: GET 2024-03-04T16:13:19.451 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.451 DEBUG:boto:Data: 2024-03-04T16:13:19.451 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.451 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.451 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.451 DEBUG:boto:Params: {} 2024-03-04T16:13:19.451 DEBUG:boto:Token: None 2024-03-04T16:13:19.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.452 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006186e9913a866331-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.454 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.455 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.456 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.456 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.456 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.456 DEBUG:boto:Method: GET 2024-03-04T16:13:19.456 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.456 DEBUG:boto:Data: 2024-03-04T16:13:19.456 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.456 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.457 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.457 DEBUG:boto:Params: {} 2024-03-04T16:13:19.457 DEBUG:boto:Token: None 2024-03-04T16:13:19.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ecff59452c85bae-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.460 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.460 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.460 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.460 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.460 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:19.460 DEBUG:boto:Data: 2024-03-04T16:13:19.460 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.460 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.461 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.461 DEBUG:boto:Params: {} 2024-03-04T16:13:19.461 DEBUG:boto:Token: None 2024-03-04T16:13:19.461 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A 2024-03-04T16:13:19.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p7pOqYwn8z9Bqab2dP8w3bJUO+U= 2024-03-04T16:13:19.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p7pOqYwn8z9Bqab2dP8w3bJUO+U=', 'Content-Length': '0'} 2024-03-04T16:13:19.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fa225189ddbeceb-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.464 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.464 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.464 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.464 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:19.464 DEBUG:boto:Data: 2024-03-04T16:13:19.464 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.464 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.465 DEBUG:boto:Params: {} 2024-03-04T16:13:19.465 DEBUG:boto:Token: None 2024-03-04T16:13:19.465 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A 2024-03-04T16:13:19.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p7pOqYwn8z9Bqab2dP8w3bJUO+U= 2024-03-04T16:13:19.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p7pOqYwn8z9Bqab2dP8w3bJUO+U=', 'Content-Length': '0'} 2024-03-04T16:13:19.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e029ef294184d4eb-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:13:19.467 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.467 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.467 DEBUG:boto:Method: GET 2024-03-04T16:13:19.467 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:19.467 DEBUG:boto:Data: 2024-03-04T16:13:19.467 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.467 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.467 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.468 DEBUG:boto:Params: {} 2024-03-04T16:13:19.468 DEBUG:boto:Token: None 2024-03-04T16:13:19.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A 2024-03-04T16:13:19.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PTUOv1mD2uIvtSbTPxOoB0/yUhQ= 2024-03-04T16:13:19.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PTUOv1mD2uIvtSbTPxOoB0/yUhQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000077dd3340d72f25d-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.470 DEBUG:boto:path=/sbnvju-10/%3A 2024-03-04T16:13:19.470 DEBUG:boto:auth_path=/sbnvju-10/%3A 2024-03-04T16:13:19.470 DEBUG:boto:Method: GET 2024-03-04T16:13:19.470 DEBUG:boto:Path: /sbnvju-10/%3A 2024-03-04T16:13:19.470 DEBUG:boto:Data: 2024-03-04T16:13:19.471 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.471 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.471 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.472 DEBUG:boto:Params: {} 2024-03-04T16:13:19.472 DEBUG:boto:Token: None 2024-03-04T16:13:19.472 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/%3A 2024-03-04T16:13:19.472 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PTUOv1mD2uIvtSbTPxOoB0/yUhQ= 2024-03-04T16:13:19.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PTUOv1mD2uIvtSbTPxOoB0/yUhQ=', 'Content-Length': '0'} 2024-03-04T16:13:19.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4cc30a0bfeb382e-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.474 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.474 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.474 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.474 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.474 DEBUG:boto:Method: GET 2024-03-04T16:13:19.475 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.475 DEBUG:boto:Data: 2024-03-04T16:13:19.475 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.475 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.475 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.475 DEBUG:boto:Params: {} 2024-03-04T16:13:19.475 DEBUG:boto:Token: None 2024-03-04T16:13:19.475 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.475 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d989de2f6e8ff7db-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.478 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.478 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.478 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.478 DEBUG:boto:path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.478 DEBUG:boto:auth_path=/sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.479 DEBUG:boto:Method: GET 2024-03-04T16:13:19.479 DEBUG:boto:Path: /sbnvju-10/?prefix=%3A 2024-03-04T16:13:19.479 DEBUG:boto:Data: 2024-03-04T16:13:19.479 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.479 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.479 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.479 DEBUG:boto:Params: {} 2024-03-04T16:13:19.479 DEBUG:boto:Token: None 2024-03-04T16:13:19.479 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.479 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71d99d5944a87d3-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.483 DEBUG:boto:b'sbnvju-10:1000false:2024-03-04T16:13:01.148Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:19.483 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.484 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.484 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.484 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.484 DEBUG:boto:Method: GET 2024-03-04T16:13:19.484 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.484 DEBUG:boto:Data: 2024-03-04T16:13:19.484 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.484 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.484 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.484 DEBUG:boto:Params: {} 2024-03-04T16:13:19.484 DEBUG:boto:Token: None 2024-03-04T16:13:19.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.485 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA= 2024-03-04T16:13:19.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA=', 'Content-Length': '0'} 2024-03-04T16:13:19.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a742a279f26a1e05-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.487 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.487 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.487 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.487 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.487 DEBUG:boto:Method: GET 2024-03-04T16:13:19.487 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.487 DEBUG:boto:Data: 2024-03-04T16:13:19.487 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.487 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.488 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.488 DEBUG:boto:Params: {} 2024-03-04T16:13:19.488 DEBUG:boto:Token: None 2024-03-04T16:13:19.488 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA= 2024-03-04T16:13:19.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA=', 'Content-Length': '0'} 2024-03-04T16:13:19.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f0d6d97f90129c-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.490 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.490 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.490 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.490 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.490 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.490 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.490 DEBUG:boto:Data: 2024-03-04T16:13:19.490 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.490 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.490 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.491 DEBUG:boto:Params: {} 2024-03-04T16:13:19.491 DEBUG:boto:Token: None 2024-03-04T16:13:19.491 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DtoMcdga+ZG7mN4foEAFDjEAvTI= 2024-03-04T16:13:19.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DtoMcdga+ZG7mN4foEAFDjEAvTI=', 'Content-Length': '0'} 2024-03-04T16:13:19.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4e7dc57b0ca2450-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.493 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.493 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.493 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.493 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.493 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.493 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.493 DEBUG:boto:Data: 2024-03-04T16:13:19.493 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.493 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.493 DEBUG:boto:Params: {} 2024-03-04T16:13:19.494 DEBUG:boto:Token: None 2024-03-04T16:13:19.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DtoMcdga+ZG7mN4foEAFDjEAvTI= 2024-03-04T16:13:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DtoMcdga+ZG7mN4foEAFDjEAvTI=', 'Content-Length': '0'} 2024-03-04T16:13:19.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cd0b2c87fd2a370-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:19.496 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.496 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.496 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.496 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.496 DEBUG:boto:Method: GET 2024-03-04T16:13:19.496 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.497 DEBUG:boto:Data: 2024-03-04T16:13:19.497 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.497 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.497 DEBUG:boto:Params: {} 2024-03-04T16:13:19.497 DEBUG:boto:Token: None 2024-03-04T16:13:19.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA= 2024-03-04T16:13:19.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA=', 'Content-Length': '0'} 2024-03-04T16:13:19.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4bee3deadcc662c-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.500 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.500 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.501 DEBUG:boto:path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.501 DEBUG:boto:auth_path=/sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.501 DEBUG:boto:Method: GET 2024-03-04T16:13:19.501 DEBUG:boto:Path: /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.501 DEBUG:boto:Data: 2024-03-04T16:13:19.501 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.501 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.501 DEBUG:boto:Params: {} 2024-03-04T16:13:19.501 DEBUG:boto:Token: None 2024-03-04T16:13:19.502 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj?versionId=null 2024-03-04T16:13:19.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA= 2024-03-04T16:13:19.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Ekyfe+emnw/HNUUUtrIX4O+ZlA=', 'Content-Length': '0'} 2024-03-04T16:13:19.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ccb3f08b402e752-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.504 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.504 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.504 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.504 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.504 DEBUG:boto:Method: GET 2024-03-04T16:13:19.504 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.504 DEBUG:boto:Data: 2024-03-04T16:13:19.504 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.505 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.505 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.505 DEBUG:boto:Params: {} 2024-03-04T16:13:19.505 DEBUG:boto:Token: None 2024-03-04T16:13:19.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.505 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007568d926176ef7a8-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.507 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.508 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.508 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.508 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.508 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.508 DEBUG:boto:Method: GET 2024-03-04T16:13:19.508 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.508 DEBUG:boto:Data: 2024-03-04T16:13:19.508 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.508 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.508 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.509 DEBUG:boto:Params: {} 2024-03-04T16:13:19.509 DEBUG:boto:Token: None 2024-03-04T16:13:19.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a78aa3fa600b5c6-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.511 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.511 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.512 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.512 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.512 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:19.512 DEBUG:boto:Data: 2024-03-04T16:13:19.512 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.512 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.512 DEBUG:boto:Params: {} 2024-03-04T16:13:19.512 DEBUG:boto:Token: None 2024-03-04T16:13:19.512 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj 2024-03-04T16:13:19.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yOOczA4Sc1cKmGFviDJch3OCU9c= 2024-03-04T16:13:19.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yOOczA4Sc1cKmGFviDJch3OCU9c=', 'Content-Length': '0'} 2024-03-04T16:13:19.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fc90ac1509febc0-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.514 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.514 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.514 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.514 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:19.515 DEBUG:boto:Data: 2024-03-04T16:13:19.515 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.515 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.515 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.515 DEBUG:boto:Params: {} 2024-03-04T16:13:19.515 DEBUG:boto:Token: None 2024-03-04T16:13:19.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj 2024-03-04T16:13:19.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yOOczA4Sc1cKmGFviDJch3OCU9c= 2024-03-04T16:13:19.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yOOczA4Sc1cKmGFviDJch3OCU9c=', 'Content-Length': '0'} 2024-03-04T16:13:19.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002be9282580f17bad-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:13:19.517 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.517 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.517 DEBUG:boto:Method: GET 2024-03-04T16:13:19.518 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:19.518 DEBUG:boto:Data: 2024-03-04T16:13:19.518 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.518 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.518 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.518 DEBUG:boto:Params: {} 2024-03-04T16:13:19.518 DEBUG:boto:Token: None 2024-03-04T16:13:19.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj 2024-03-04T16:13:19.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fC6B/GuMNfBO/Zda/KYiMYX8Se8= 2024-03-04T16:13:19.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fC6B/GuMNfBO/Zda/KYiMYX8Se8=', 'Content-Length': '0'} 2024-03-04T16:13:19.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae09e2635db71af8-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.520 DEBUG:boto:path=/sbnvju-10/_myobj 2024-03-04T16:13:19.520 DEBUG:boto:auth_path=/sbnvju-10/_myobj 2024-03-04T16:13:19.520 DEBUG:boto:Method: GET 2024-03-04T16:13:19.520 DEBUG:boto:Path: /sbnvju-10/_myobj 2024-03-04T16:13:19.521 DEBUG:boto:Data: 2024-03-04T16:13:19.521 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.521 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.521 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.521 DEBUG:boto:Params: {} 2024-03-04T16:13:19.521 DEBUG:boto:Token: None 2024-03-04T16:13:19.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/_myobj 2024-03-04T16:13:19.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fC6B/GuMNfBO/Zda/KYiMYX8Se8= 2024-03-04T16:13:19.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fC6B/GuMNfBO/Zda/KYiMYX8Se8=', 'Content-Length': '0'} 2024-03-04T16:13:19.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a14dd969f6c487a3-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.523 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.523 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.523 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.523 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.523 DEBUG:boto:Method: GET 2024-03-04T16:13:19.523 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.523 DEBUG:boto:Data: 2024-03-04T16:13:19.524 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.524 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.524 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.524 DEBUG:boto:Params: {} 2024-03-04T16:13:19.524 DEBUG:boto:Token: None 2024-03-04T16:13:19.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea249854688b083b-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.526 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.527 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.527 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.527 DEBUG:boto:path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.527 DEBUG:boto:auth_path=/sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.527 DEBUG:boto:Method: GET 2024-03-04T16:13:19.527 DEBUG:boto:Path: /sbnvju-10/?prefix=_myobj 2024-03-04T16:13:19.527 DEBUG:boto:Data: 2024-03-04T16:13:19.527 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.527 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.527 DEBUG:boto:Params: {} 2024-03-04T16:13:19.528 DEBUG:boto:Token: None 2024-03-04T16:13:19.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b758c3a76c05ce-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.530 DEBUG:boto:b'sbnvju-10_myobj1000false_myobj2024-03-04T16:13:01.130Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:19.531 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.531 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.531 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.531 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.531 DEBUG:boto:Method: GET 2024-03-04T16:13:19.531 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.531 DEBUG:boto:Data: 2024-03-04T16:13:19.531 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.531 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.532 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.532 DEBUG:boto:Params: {} 2024-03-04T16:13:19.532 DEBUG:boto:Token: None 2024-03-04T16:13:19.532 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.532 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA= 2024-03-04T16:13:19.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA=', 'Content-Length': '0'} 2024-03-04T16:13:19.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000755059fa8746f27c-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.534 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.534 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.534 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.534 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.534 DEBUG:boto:Method: GET 2024-03-04T16:13:19.535 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.535 DEBUG:boto:Data: 2024-03-04T16:13:19.535 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.535 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.535 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.535 DEBUG:boto:Params: {} 2024-03-04T16:13:19.535 DEBUG:boto:Token: None 2024-03-04T16:13:19.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.535 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA= 2024-03-04T16:13:19.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA=', 'Content-Length': '0'} 2024-03-04T16:13:19.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5d6b5d58f202ba2-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.537 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.538 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.538 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.538 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.538 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.538 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.538 DEBUG:boto:Data: 2024-03-04T16:13:19.538 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.538 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.538 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.538 DEBUG:boto:Params: {} 2024-03-04T16:13:19.538 DEBUG:boto:Token: None 2024-03-04T16:13:19.539 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u57YGsN/Ggk8oYF11U3b1udXZjw= 2024-03-04T16:13:19.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u57YGsN/Ggk8oYF11U3b1udXZjw=', 'Content-Length': '0'} 2024-03-04T16:13:19.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000458d79b3802e9e9-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.540 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.540 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.541 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.541 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.541 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.541 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.541 DEBUG:boto:Data: 2024-03-04T16:13:19.541 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.541 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.541 DEBUG:boto:Params: {} 2024-03-04T16:13:19.541 DEBUG:boto:Token: None 2024-03-04T16:13:19.541 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u57YGsN/Ggk8oYF11U3b1udXZjw= 2024-03-04T16:13:19.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u57YGsN/Ggk8oYF11U3b1udXZjw=', 'Content-Length': '0'} 2024-03-04T16:13:19.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000798d7de88a5623a1-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:19.544 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.544 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.544 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.544 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.544 DEBUG:boto:Method: GET 2024-03-04T16:13:19.544 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.544 DEBUG:boto:Data: 2024-03-04T16:13:19.544 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.544 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.544 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.545 DEBUG:boto:Params: {} 2024-03-04T16:13:19.545 DEBUG:boto:Token: None 2024-03-04T16:13:19.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA= 2024-03-04T16:13:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA=', 'Content-Length': '0'} 2024-03-04T16:13:19.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fb3a9fcf9edf913-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.592 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.592 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.592 DEBUG:boto:path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.593 DEBUG:boto:auth_path=/sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.593 DEBUG:boto:Method: GET 2024-03-04T16:13:19.593 DEBUG:boto:Path: /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.593 DEBUG:boto:Data: 2024-03-04T16:13:19.593 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.593 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.593 DEBUG:boto:Params: {} 2024-03-04T16:13:19.593 DEBUG:boto:Token: None 2024-03-04T16:13:19.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj?versionId=null 2024-03-04T16:13:19.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA= 2024-03-04T16:13:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fKlZHKj0QoZihRVD+4fgYUAzgUA=', 'Content-Length': '0'} 2024-03-04T16:13:19.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068659135a6ac0d4a-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.596 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.596 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.596 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.596 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.597 DEBUG:boto:Method: GET 2024-03-04T16:13:19.597 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.597 DEBUG:boto:Data: 2024-03-04T16:13:19.597 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.597 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.597 DEBUG:boto:Params: {} 2024-03-04T16:13:19.597 DEBUG:boto:Token: None 2024-03-04T16:13:19.597 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000071b1e9cee817952-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.600 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.600 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.600 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.600 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.601 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.601 DEBUG:boto:Method: GET 2024-03-04T16:13:19.601 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.601 DEBUG:boto:Data: 2024-03-04T16:13:19.601 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.601 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.601 DEBUG:boto:Params: {} 2024-03-04T16:13:19.602 DEBUG:boto:Token: None 2024-03-04T16:13:19.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003803c6ae9fc376b1-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.604 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.604 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.604 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.604 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.605 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:19.605 DEBUG:boto:Data: 2024-03-04T16:13:19.605 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.605 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.605 DEBUG:boto:Params: {} 2024-03-04T16:13:19.605 DEBUG:boto:Token: None 2024-03-04T16:13:19.605 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj 2024-03-04T16:13:19.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:25TDK+iK+P4DwpddEfR6JEJF9qo= 2024-03-04T16:13:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:25TDK+iK+P4DwpddEfR6JEJF9qo=', 'Content-Length': '0'} 2024-03-04T16:13:19.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b0fb2395abd4e87-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.607 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.607 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.607 DEBUG:boto:Method: HEAD 2024-03-04T16:13:19.608 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:19.608 DEBUG:boto:Data: 2024-03-04T16:13:19.608 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.608 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.608 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.608 DEBUG:boto:Params: {} 2024-03-04T16:13:19.608 DEBUG:boto:Token: None 2024-03-04T16:13:19.608 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj 2024-03-04T16:13:19.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:25TDK+iK+P4DwpddEfR6JEJF9qo= 2024-03-04T16:13:19.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:25TDK+iK+P4DwpddEfR6JEJF9qo=', 'Content-Length': '0'} 2024-03-04T16:13:19.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aa41994ac8807a4-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:13:19.610 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.611 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.611 DEBUG:boto:Method: GET 2024-03-04T16:13:19.611 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:19.611 DEBUG:boto:Data: 2024-03-04T16:13:19.611 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.611 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.611 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.611 DEBUG:boto:Params: {} 2024-03-04T16:13:19.611 DEBUG:boto:Token: None 2024-03-04T16:13:19.611 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj 2024-03-04T16:13:19.612 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I5AgO3TsMbeNz1QGU/sniEKnr4E= 2024-03-04T16:13:19.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I5AgO3TsMbeNz1QGU/sniEKnr4E=', 'Content-Length': '0'} 2024-03-04T16:13:19.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-10:fc294220-5038-4f25-aeee-cac2c5838729.4341.19'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b84c766ce6b6ccbd-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.614 DEBUG:boto:path=/sbnvju-10/myobj 2024-03-04T16:13:19.614 DEBUG:boto:auth_path=/sbnvju-10/myobj 2024-03-04T16:13:19.614 DEBUG:boto:Method: GET 2024-03-04T16:13:19.614 DEBUG:boto:Path: /sbnvju-10/myobj 2024-03-04T16:13:19.614 DEBUG:boto:Data: 2024-03-04T16:13:19.614 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.614 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.614 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.614 DEBUG:boto:Params: {} 2024-03-04T16:13:19.614 DEBUG:boto:Token: None 2024-03-04T16:13:19.615 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/myobj 2024-03-04T16:13:19.615 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I5AgO3TsMbeNz1QGU/sniEKnr4E= 2024-03-04T16:13:19.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I5AgO3TsMbeNz1QGU/sniEKnr4E=', 'Content-Length': '0'} 2024-03-04T16:13:19.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038ceacdd2e340e18-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.660 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.660 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.660 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.661 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.661 DEBUG:boto:Method: GET 2024-03-04T16:13:19.661 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.661 DEBUG:boto:Data: 2024-03-04T16:13:19.661 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.661 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:13:19.661 DEBUG:boto:Port: 8001 2024-03-04T16:13:19.661 DEBUG:boto:Params: {} 2024-03-04T16:13:19.661 DEBUG:boto:Token: None 2024-03-04T16:13:19.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000362aae4dff890767-0065e5f31f-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.664 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.665 DEBUG:boto:path=/sbnvju-10/ 2024-03-04T16:13:19.665 DEBUG:boto:auth_path=/sbnvju-10/ 2024-03-04T16:13:19.665 DEBUG:boto:path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.665 DEBUG:boto:auth_path=/sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.665 DEBUG:boto:Method: GET 2024-03-04T16:13:19.665 DEBUG:boto:Path: /sbnvju-10/?prefix=myobj 2024-03-04T16:13:19.665 DEBUG:boto:Data: 2024-03-04T16:13:19.665 DEBUG:boto:Headers: {} 2024-03-04T16:13:19.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:13:19.665 DEBUG:boto:Port: 8000 2024-03-04T16:13:19.665 DEBUG:boto:Params: {} 2024-03-04T16:13:19.666 DEBUG:boto:Token: None 2024-03-04T16:13:19.666 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:13:19 GMT /sbnvju-10/ 2024-03-04T16:13:19.666 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E= 2024-03-04T16:13:19.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:13:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NDr37KU+SC8mQTpQD7GlF23bA/E=', 'Content-Length': '0'} 2024-03-04T16:13:19.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004527f6a0f0c2fb12-0065e5f31f-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:13:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:13:19.668 DEBUG:boto:b'sbnvju-10myobj1000falsemyobj2024-03-04T16:13:01.115Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:13:19.668 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-10 zones={test-zone3, test-zone1} 2024-03-04T16:13:19.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:19.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:13:19.669 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:13:19.723 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:13:19.723 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:13:19.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:19.772+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:24.773+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:29.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:29.774+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:34.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:34.774+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:39.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:39.775+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:44.775+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:49.776+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:54.777+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:13:59.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:13:59.777+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:04.778+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:09.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:09.779+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:14.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:14.779+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:19.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:19.780+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:24.780+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:29.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:29.781+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:34.782+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:39.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:39.783+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:44.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:44.784+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:49.784+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:54.785+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:14:59.786+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:04.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:04.786+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:09.787+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:14.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:14.788+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:19.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:19.788+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:24.789+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:29.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:29.790+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:34.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:34.790+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:39.791+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:44.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:44.792+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:49.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:49.792+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:54.793+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:15:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:15:59.794+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:04.795+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:09.796+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:14.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:14.796+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:19.797+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:24.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:24.797+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:29.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:29.798+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:34.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:34.799+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:39.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:39.799+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:44.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:44.800+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:49.800+0000 7fa58b90d800 1 waiting to reach incremental sync.. 2024-03-04T16:16:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:54.802+0000 7fa58b90d800 1 bucket sync caught up with source: 2024-03-04T16:16:54.803 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.21.6, , 00000000001.18.6, , 00000000001.13.6, 00000000002.15.6, 00000000001.7.6, 00000000001.5.6, 00000000001.11.6, ] 2024-03-04T16:16:54.803 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.21.6, , 00000000001.18.6, , 00000000001.13.6, 00000000002.15.6, 00000000001.7.6, 00000000001.5.6, 00000000001.11.6, ] 2024-03-04T16:16:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:54.802+0000 7fa58b90d800 0 bucket checkpoint complete 2024-03-04T16:16:54.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-11 zones={test-zone1, test-zone2} 2024-03-04T16:16:54.828 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.828 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.828 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.828 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:16:54.828 DEBUG:boto:Data: 2024-03-04T16:16:54.828 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.829 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.829 DEBUG:boto:Params: {} 2024-03-04T16:16:54.829 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:16:54.830 DEBUG:boto:Token: None 2024-03-04T16:16:54.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Sr+H7DBJHUnFpv+0dylsleQR2lk= 2024-03-04T16:16:54.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Sr+H7DBJHUnFpv+0dylsleQR2lk=', 'Content-Length': '0'} 2024-03-04T16:16:54.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013d9d904b706a075-0065e5f3f6-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.836 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.836 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.836 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.836 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:16:54.836 DEBUG:boto:Data: 2024-03-04T16:16:54.837 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.837 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.837 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.837 DEBUG:boto:Params: {} 2024-03-04T16:16:54.837 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:16:54.837 DEBUG:boto:Token: None 2024-03-04T16:16:54.838 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Sr+H7DBJHUnFpv+0dylsleQR2lk= 2024-03-04T16:16:54.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Sr+H7DBJHUnFpv+0dylsleQR2lk=', 'Content-Length': '0'} 2024-03-04T16:16:54.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa9595177a8bf276-0065e5f3f6-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:16:54.843 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.843 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.843 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.843 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.843 DEBUG:boto:Method: GET 2024-03-04T16:16:54.843 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.843 DEBUG:boto:Data: 2024-03-04T16:16:54.844 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.844 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.844 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.844 DEBUG:boto:Params: {} 2024-03-04T16:16:54.844 DEBUG:boto:Token: None 2024-03-04T16:16:54.845 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/?versions 2024-03-04T16:16:54.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U= 2024-03-04T16:16:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U=', 'Content-Length': '0'} 2024-03-04T16:16:54.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9461cf9cf1ac770-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.848 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.849 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:54.849 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:54.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:16:54.850 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.850 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.850 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.850 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.850 DEBUG:boto:Method: GET 2024-03-04T16:16:54.851 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.851 DEBUG:boto:Data: 2024-03-04T16:16:54.851 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.851 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.851 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.851 DEBUG:boto:Params: {} 2024-03-04T16:16:54.851 DEBUG:boto:Token: None 2024-03-04T16:16:54.851 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/?versions 2024-03-04T16:16:54.851 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U= 2024-03-04T16:16:54.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U=', 'Content-Length': '0'} 2024-03-04T16:16:54.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900a5c2170262579-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.854 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:54.855 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:54.856 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.856 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.856 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.856 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.856 DEBUG:boto:Method: GET 2024-03-04T16:16:54.856 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.856 DEBUG:boto:Data: 2024-03-04T16:16:54.856 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.857 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.857 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.857 DEBUG:boto:Params: {} 2024-03-04T16:16:54.857 DEBUG:boto:Token: None 2024-03-04T16:16:54.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/?versions 2024-03-04T16:16:54.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U= 2024-03-04T16:16:54.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U=', 'Content-Length': '0'} 2024-03-04T16:16:54.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773e74dceddca170-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.860 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.860 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.861 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.861 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.861 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.861 DEBUG:boto:Method: GET 2024-03-04T16:16:54.861 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:54.861 DEBUG:boto:Data: 2024-03-04T16:16:54.861 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.861 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.861 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.861 DEBUG:boto:Params: {} 2024-03-04T16:16:54.862 DEBUG:boto:Token: None 2024-03-04T16:16:54.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/?versions 2024-03-04T16:16:54.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U= 2024-03-04T16:16:54.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMnZvqSSiIBH8WBIdolVk3iF0U=', 'Content-Length': '0'} 2024-03-04T16:16:54.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4adb4ce9dbfbc4d-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.864 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:54.865 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.865 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.865 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.866 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.866 DEBUG:boto:Method: GET 2024-03-04T16:16:54.866 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.866 DEBUG:boto:Data: 2024-03-04T16:16:54.866 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.866 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.866 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.866 DEBUG:boto:Params: {} 2024-03-04T16:16:54.866 DEBUG:boto:Token: None 2024-03-04T16:16:54.866 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.866 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4= 2024-03-04T16:16:54.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4=', 'Content-Length': '0'} 2024-03-04T16:16:54.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099974cf6d2210d7c-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.868 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.869 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.869 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.869 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.869 DEBUG:boto:Method: GET 2024-03-04T16:16:54.869 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.869 DEBUG:boto:Data: 2024-03-04T16:16:54.869 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.869 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.869 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.870 DEBUG:boto:Params: {} 2024-03-04T16:16:54.870 DEBUG:boto:Token: None 2024-03-04T16:16:54.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.870 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4= 2024-03-04T16:16:54.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4=', 'Content-Length': '0'} 2024-03-04T16:16:54.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090afbca54057984b-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.872 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.873 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.873 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.873 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.873 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.873 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.873 DEBUG:boto:Data: 2024-03-04T16:16:54.873 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.874 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.874 DEBUG:boto:Params: {} 2024-03-04T16:16:54.874 DEBUG:boto:Token: None 2024-03-04T16:16:54.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.875 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:slv0V9seIOyeLiGENNr7F7YBk9g= 2024-03-04T16:16:54.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:slv0V9seIOyeLiGENNr7F7YBk9g=', 'Content-Length': '0'} 2024-03-04T16:16:54.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001386e006a9a06b0e-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.877 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.877 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.877 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.877 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.877 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.877 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.877 DEBUG:boto:Data: 2024-03-04T16:16:54.877 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.878 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.878 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.878 DEBUG:boto:Params: {} 2024-03-04T16:16:54.878 DEBUG:boto:Token: None 2024-03-04T16:16:54.878 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:slv0V9seIOyeLiGENNr7F7YBk9g= 2024-03-04T16:16:54.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:slv0V9seIOyeLiGENNr7F7YBk9g=', 'Content-Length': '0'} 2024-03-04T16:16:54.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da1799d31e8d21d5-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:54.880 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.881 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.881 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.881 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.881 DEBUG:boto:Method: GET 2024-03-04T16:16:54.881 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.881 DEBUG:boto:Data: 2024-03-04T16:16:54.881 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.881 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.881 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.881 DEBUG:boto:Params: {} 2024-03-04T16:16:54.881 DEBUG:boto:Token: None 2024-03-04T16:16:54.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.882 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4= 2024-03-04T16:16:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4=', 'Content-Length': '0'} 2024-03-04T16:16:54.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac26dada253b8ec9-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.884 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.884 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.884 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.884 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.885 DEBUG:boto:Method: GET 2024-03-04T16:16:54.885 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.885 DEBUG:boto:Data: 2024-03-04T16:16:54.885 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.885 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.885 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.885 DEBUG:boto:Params: {} 2024-03-04T16:16:54.885 DEBUG:boto:Token: None 2024-03-04T16:16:54.885 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:54.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4= 2024-03-04T16:16:54.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0XZal8YRncqhpURGdRvZqY0nQP4=', 'Content-Length': '0'} 2024-03-04T16:16:54.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2626c4fade365c4-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.928 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.928 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.928 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.928 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.928 DEBUG:boto:Method: GET 2024-03-04T16:16:54.928 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:54.929 DEBUG:boto:Data: 2024-03-04T16:16:54.929 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.929 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.929 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.929 DEBUG:boto:Params: {} 2024-03-04T16:16:54.929 DEBUG:boto:Token: None 2024-03-04T16:16:54.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8= 2024-03-04T16:16:54.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8=', 'Content-Length': '0'} 2024-03-04T16:16:54.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec120aa7f8aed0d8-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.933 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.933 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.933 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.933 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.933 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.934 DEBUG:boto:Method: GET 2024-03-04T16:16:54.934 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:54.934 DEBUG:boto:Data: 2024-03-04T16:16:54.934 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.934 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.934 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.934 DEBUG:boto:Params: {} 2024-03-04T16:16:54.935 DEBUG:boto:Token: None 2024-03-04T16:16:54.935 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.935 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8= 2024-03-04T16:16:54.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8=', 'Content-Length': '0'} 2024-03-04T16:16:54.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e3c3cb4f09c96d-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.938 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.938 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.938 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.938 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.938 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:54.938 DEBUG:boto:Data: 2024-03-04T16:16:54.938 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.939 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.939 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.939 DEBUG:boto:Params: {} 2024-03-04T16:16:54.939 DEBUG:boto:Token: None 2024-03-04T16:16:54.939 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26 2024-03-04T16:16:54.939 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lcDp18q7iIX2wQC6xIx0vc7vINA= 2024-03-04T16:16:54.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lcDp18q7iIX2wQC6xIx0vc7vINA=', 'Content-Length': '0'} 2024-03-04T16:16:54.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dca78f7386f04d8-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.941 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.941 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.941 DEBUG:boto:Method: HEAD 2024-03-04T16:16:54.942 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:54.942 DEBUG:boto:Data: 2024-03-04T16:16:54.942 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.942 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.942 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.942 DEBUG:boto:Params: {} 2024-03-04T16:16:54.942 DEBUG:boto:Token: None 2024-03-04T16:16:54.942 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26 2024-03-04T16:16:54.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lcDp18q7iIX2wQC6xIx0vc7vINA= 2024-03-04T16:16:54.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lcDp18q7iIX2wQC6xIx0vc7vINA=', 'Content-Length': '0'} 2024-03-04T16:16:54.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c60be6c157634e-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:54.945 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.945 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.945 DEBUG:boto:Method: GET 2024-03-04T16:16:54.945 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:54.945 DEBUG:boto:Data: 2024-03-04T16:16:54.945 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.945 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.945 DEBUG:boto:Params: {} 2024-03-04T16:16:54.946 DEBUG:boto:Token: None 2024-03-04T16:16:54.946 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26 2024-03-04T16:16:54.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AJ51p2BZcMw4jgXIdpBJlfEOayU= 2024-03-04T16:16:54.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AJ51p2BZcMw4jgXIdpBJlfEOayU=', 'Content-Length': '0'} 2024-03-04T16:16:54.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa79b29a8d956c36-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.988 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:54.988 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:54.988 DEBUG:boto:Method: GET 2024-03-04T16:16:54.988 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:54.988 DEBUG:boto:Data: 2024-03-04T16:16:54.989 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.989 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.989 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.989 DEBUG:boto:Params: {} 2024-03-04T16:16:54.989 DEBUG:boto:Token: None 2024-03-04T16:16:54.989 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/%26 2024-03-04T16:16:54.989 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AJ51p2BZcMw4jgXIdpBJlfEOayU= 2024-03-04T16:16:54.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AJ51p2BZcMw4jgXIdpBJlfEOayU=', 'Content-Length': '0'} 2024-03-04T16:16:54.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001676585f0d6e8126-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.991 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.991 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.992 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.992 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.992 DEBUG:boto:Method: GET 2024-03-04T16:16:54.992 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:54.992 DEBUG:boto:Data: 2024-03-04T16:16:54.992 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.992 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:54.992 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.993 DEBUG:boto:Params: {} 2024-03-04T16:16:54.993 DEBUG:boto:Token: None 2024-03-04T16:16:54.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.993 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8= 2024-03-04T16:16:54.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8=', 'Content-Length': '0'} 2024-03-04T16:16:54.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000567f58a3ab2ceaba-0065e5f3f6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:54.996 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:54.996 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:54.996 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:54.996 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.996 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:54.997 DEBUG:boto:Method: GET 2024-03-04T16:16:54.997 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:54.997 DEBUG:boto:Data: 2024-03-04T16:16:54.997 DEBUG:boto:Headers: {} 2024-03-04T16:16:54.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:54.998 DEBUG:boto:Port: 8000 2024-03-04T16:16:54.998 DEBUG:boto:Params: {} 2024-03-04T16:16:54.998 DEBUG:boto:Token: None 2024-03-04T16:16:54.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:54 GMT /sbnvju-11/ 2024-03-04T16:16:54.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8= 2024-03-04T16:16:54.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R4ul55zk38T5jJ4PyW8Qxbc6lT8=', 'Content-Length': '0'} 2024-03-04T16:16:55.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e00921b571201f-0065e5f3f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.001 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:55.001 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.001 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.001 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.001 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.001 DEBUG:boto:Method: GET 2024-03-04T16:16:55.001 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.002 DEBUG:boto:Data: 2024-03-04T16:16:55.002 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.002 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.002 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.002 DEBUG:boto:Params: {} 2024-03-04T16:16:55.002 DEBUG:boto:Token: None 2024-03-04T16:16:55.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE= 2024-03-04T16:16:55.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE=', 'Content-Length': '0'} 2024-03-04T16:16:55.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000122fcb6bb6eb311-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.004 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.005 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.005 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.005 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.005 DEBUG:boto:Method: GET 2024-03-04T16:16:55.005 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.005 DEBUG:boto:Data: 2024-03-04T16:16:55.005 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.005 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.005 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.006 DEBUG:boto:Params: {} 2024-03-04T16:16:55.006 DEBUG:boto:Token: None 2024-03-04T16:16:55.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.006 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE= 2024-03-04T16:16:55.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE=', 'Content-Length': '0'} 2024-03-04T16:16:55.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bade2e67b60eec3-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.048 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.048 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.048 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.048 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.049 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.049 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.049 DEBUG:boto:Data: 2024-03-04T16:16:55.049 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.049 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.049 DEBUG:boto:Params: {} 2024-03-04T16:16:55.049 DEBUG:boto:Token: None 2024-03-04T16:16:55.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ef8/W5dfFWSRHUXCRxh6qml/LC4= 2024-03-04T16:16:55.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ef8/W5dfFWSRHUXCRxh6qml/LC4=', 'Content-Length': '0'} 2024-03-04T16:16:55.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000700e4aff3483e5e4-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.052 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.052 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.052 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.052 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.052 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.052 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.052 DEBUG:boto:Data: 2024-03-04T16:16:55.052 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.052 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.052 DEBUG:boto:Params: {} 2024-03-04T16:16:55.053 DEBUG:boto:Token: None 2024-03-04T16:16:55.053 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ef8/W5dfFWSRHUXCRxh6qml/LC4= 2024-03-04T16:16:55.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ef8/W5dfFWSRHUXCRxh6qml/LC4=', 'Content-Length': '0'} 2024-03-04T16:16:55.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed1e65c718fa627-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:55.056 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.056 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.056 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.056 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.056 DEBUG:boto:Method: GET 2024-03-04T16:16:55.056 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.056 DEBUG:boto:Data: 2024-03-04T16:16:55.056 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.056 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.057 DEBUG:boto:Params: {} 2024-03-04T16:16:55.057 DEBUG:boto:Token: None 2024-03-04T16:16:55.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.057 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE= 2024-03-04T16:16:55.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE=', 'Content-Length': '0'} 2024-03-04T16:16:55.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8bbef1391d93df-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.100 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.100 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.100 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.100 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:55.101 DEBUG:boto:Method: GET 2024-03-04T16:16:55.101 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.101 DEBUG:boto:Data: 2024-03-04T16:16:55.101 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.101 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.102 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.102 DEBUG:boto:Params: {} 2024-03-04T16:16:55.102 DEBUG:boto:Token: None 2024-03-04T16:16:55.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:55.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE= 2024-03-04T16:16:55.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/49xBTv8E9EG0wZIykV/HWHZhE=', 'Content-Length': '0'} 2024-03-04T16:16:55.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080c8f89c115200bc-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.105 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.105 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.105 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.106 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.106 DEBUG:boto:Method: GET 2024-03-04T16:16:55.106 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:55.106 DEBUG:boto:Data: 2024-03-04T16:16:55.106 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.106 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.106 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.106 DEBUG:boto:Params: {} 2024-03-04T16:16:55.106 DEBUG:boto:Token: None 2024-03-04T16:16:55.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.107 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068bb889b3d1e8179-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.109 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.110 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.110 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.110 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.110 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.110 DEBUG:boto:Method: GET 2024-03-04T16:16:55.110 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:55.110 DEBUG:boto:Data: 2024-03-04T16:16:55.110 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.110 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.111 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.111 DEBUG:boto:Params: {} 2024-03-04T16:16:55.111 DEBUG:boto:Token: None 2024-03-04T16:16:55.111 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.111 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf866e99c77ed6b-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.114 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.114 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.114 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.114 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.115 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:55.115 DEBUG:boto:Data: 2024-03-04T16:16:55.115 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.115 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.115 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.115 DEBUG:boto:Params: {} 2024-03-04T16:16:55.115 DEBUG:boto:Token: None 2024-03-04T16:16:55.115 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/. 2024-03-04T16:16:55.116 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZCczhWBOyNR+8SRWGfY0HHiq78A= 2024-03-04T16:16:55.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZCczhWBOyNR+8SRWGfY0HHiq78A=', 'Content-Length': '0'} 2024-03-04T16:16:55.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ddb68aee7fe2fdc-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.118 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.118 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.118 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.118 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:55.118 DEBUG:boto:Data: 2024-03-04T16:16:55.118 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.118 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.118 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.118 DEBUG:boto:Params: {} 2024-03-04T16:16:55.119 DEBUG:boto:Token: None 2024-03-04T16:16:55.119 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/. 2024-03-04T16:16:55.119 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZCczhWBOyNR+8SRWGfY0HHiq78A= 2024-03-04T16:16:55.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZCczhWBOyNR+8SRWGfY0HHiq78A=', 'Content-Length': '0'} 2024-03-04T16:16:55.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000663de45739c18cb-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:55.121 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.121 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.121 DEBUG:boto:Method: GET 2024-03-04T16:16:55.121 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:55.121 DEBUG:boto:Data: 2024-03-04T16:16:55.121 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.121 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.122 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.122 DEBUG:boto:Params: {} 2024-03-04T16:16:55.122 DEBUG:boto:Token: None 2024-03-04T16:16:55.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/. 2024-03-04T16:16:55.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Gxn5ti74fyRH1iElNzrdM0IkaYs= 2024-03-04T16:16:55.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Gxn5ti74fyRH1iElNzrdM0IkaYs=', 'Content-Length': '0'} 2024-03-04T16:16:55.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd7c26fd4ddad3ec-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.164 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:55.164 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:55.164 DEBUG:boto:Method: GET 2024-03-04T16:16:55.165 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:55.165 DEBUG:boto:Data: 2024-03-04T16:16:55.165 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.165 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.165 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.165 DEBUG:boto:Params: {} 2024-03-04T16:16:55.165 DEBUG:boto:Token: None 2024-03-04T16:16:55.165 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/. 2024-03-04T16:16:55.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Gxn5ti74fyRH1iElNzrdM0IkaYs= 2024-03-04T16:16:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Gxn5ti74fyRH1iElNzrdM0IkaYs=', 'Content-Length': '0'} 2024-03-04T16:16:55.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e8b41fef706e147-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.168 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.168 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.168 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.168 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.168 DEBUG:boto:Method: GET 2024-03-04T16:16:55.168 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:55.168 DEBUG:boto:Data: 2024-03-04T16:16:55.168 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.168 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.168 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.168 DEBUG:boto:Params: {} 2024-03-04T16:16:55.169 DEBUG:boto:Token: None 2024-03-04T16:16:55.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c1023539eec989-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.172 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.172 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.172 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.172 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.173 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:55.173 DEBUG:boto:Method: GET 2024-03-04T16:16:55.173 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:55.173 DEBUG:boto:Data: 2024-03-04T16:16:55.173 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.173 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.173 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.173 DEBUG:boto:Params: {} 2024-03-04T16:16:55.173 DEBUG:boto:Token: None 2024-03-04T16:16:55.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5d42303fd4b6c9-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.177 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:55.177 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.177 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.177 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.178 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.178 DEBUG:boto:Method: GET 2024-03-04T16:16:55.178 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.178 DEBUG:boto:Data: 2024-03-04T16:16:55.178 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.178 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.178 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.178 DEBUG:boto:Params: {} 2024-03-04T16:16:55.178 DEBUG:boto:Token: None 2024-03-04T16:16:55.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.179 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8= 2024-03-04T16:16:55.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8=', 'Content-Length': '0'} 2024-03-04T16:16:55.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f2fb726ae85b0fa-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.181 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.181 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.181 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.181 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.181 DEBUG:boto:Method: GET 2024-03-04T16:16:55.181 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.181 DEBUG:boto:Data: 2024-03-04T16:16:55.182 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.182 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.182 DEBUG:boto:Params: {} 2024-03-04T16:16:55.182 DEBUG:boto:Token: None 2024-03-04T16:16:55.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8= 2024-03-04T16:16:55.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8=', 'Content-Length': '0'} 2024-03-04T16:16:55.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e5b7c3697037448-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.184 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.185 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.185 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.185 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.185 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.185 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.185 DEBUG:boto:Data: 2024-03-04T16:16:55.185 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.185 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.185 DEBUG:boto:Params: {} 2024-03-04T16:16:55.185 DEBUG:boto:Token: None 2024-03-04T16:16:55.186 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Eux+HhIsHsUAm7pvoPxRPmDrbkc= 2024-03-04T16:16:55.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Eux+HhIsHsUAm7pvoPxRPmDrbkc=', 'Content-Length': '0'} 2024-03-04T16:16:55.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8ab328a1c5c9836-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.188 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.188 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.188 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.188 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.188 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.188 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.188 DEBUG:boto:Data: 2024-03-04T16:16:55.188 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.189 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.189 DEBUG:boto:Params: {} 2024-03-04T16:16:55.189 DEBUG:boto:Token: None 2024-03-04T16:16:55.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Eux+HhIsHsUAm7pvoPxRPmDrbkc= 2024-03-04T16:16:55.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Eux+HhIsHsUAm7pvoPxRPmDrbkc=', 'Content-Length': '0'} 2024-03-04T16:16:55.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004072270d0afcbcc-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:55.191 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.191 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.191 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.192 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.192 DEBUG:boto:Method: GET 2024-03-04T16:16:55.192 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.192 DEBUG:boto:Data: 2024-03-04T16:16:55.192 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.192 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.192 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.192 DEBUG:boto:Params: {} 2024-03-04T16:16:55.192 DEBUG:boto:Token: None 2024-03-04T16:16:55.192 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.193 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8= 2024-03-04T16:16:55.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8=', 'Content-Length': '0'} 2024-03-04T16:16:55.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040fd5001e92b2e30-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.240 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.240 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.240 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.240 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:55.241 DEBUG:boto:Method: GET 2024-03-04T16:16:55.241 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.241 DEBUG:boto:Data: 2024-03-04T16:16:55.241 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.241 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.241 DEBUG:boto:Params: {} 2024-03-04T16:16:55.241 DEBUG:boto:Token: None 2024-03-04T16:16:55.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:55.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8= 2024-03-04T16:16:55.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mi/I0ekCSYJvaUV0tBYboGYy9c8=', 'Content-Length': '0'} 2024-03-04T16:16:55.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000915972e7c9aa4fb2-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.244 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.244 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.244 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.244 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.244 DEBUG:boto:Method: GET 2024-03-04T16:16:55.244 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:55.245 DEBUG:boto:Data: 2024-03-04T16:16:55.245 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.245 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.245 DEBUG:boto:Params: {} 2024-03-04T16:16:55.245 DEBUG:boto:Token: None 2024-03-04T16:16:55.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083977bf88e3c51da-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.292 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.292 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.293 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.293 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.293 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.293 DEBUG:boto:Method: GET 2024-03-04T16:16:55.293 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:55.293 DEBUG:boto:Data: 2024-03-04T16:16:55.293 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.293 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.293 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.294 DEBUG:boto:Params: {} 2024-03-04T16:16:55.294 DEBUG:boto:Token: None 2024-03-04T16:16:55.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ce8693f2947006-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.297 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.297 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.297 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.298 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.298 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:55.298 DEBUG:boto:Data: 2024-03-04T16:16:55.298 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.298 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.298 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.298 DEBUG:boto:Params: {} 2024-03-04T16:16:55.298 DEBUG:boto:Token: None 2024-03-04T16:16:55.298 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.. 2024-03-04T16:16:55.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pDqghEmYn36ZWCJOys2H79RfgWA= 2024-03-04T16:16:55.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pDqghEmYn36ZWCJOys2H79RfgWA=', 'Content-Length': '0'} 2024-03-04T16:16:55.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a54bc0c60b8dc1f9-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.301 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.301 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.301 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.301 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:55.301 DEBUG:boto:Data: 2024-03-04T16:16:55.301 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.302 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.302 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.302 DEBUG:boto:Params: {} 2024-03-04T16:16:55.302 DEBUG:boto:Token: None 2024-03-04T16:16:55.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.. 2024-03-04T16:16:55.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pDqghEmYn36ZWCJOys2H79RfgWA= 2024-03-04T16:16:55.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pDqghEmYn36ZWCJOys2H79RfgWA=', 'Content-Length': '0'} 2024-03-04T16:16:55.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afe5ad185ffa5432-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:55.304 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.305 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.305 DEBUG:boto:Method: GET 2024-03-04T16:16:55.305 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:55.305 DEBUG:boto:Data: 2024-03-04T16:16:55.305 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.305 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.305 DEBUG:boto:Params: {} 2024-03-04T16:16:55.305 DEBUG:boto:Token: None 2024-03-04T16:16:55.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.. 2024-03-04T16:16:55.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FonIW4bcBaG1mZWQsT12jlp0fTs= 2024-03-04T16:16:55.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FonIW4bcBaG1mZWQsT12jlp0fTs=', 'Content-Length': '0'} 2024-03-04T16:16:55.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb4b3d910444c266-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.348 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:55.348 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:55.349 DEBUG:boto:Method: GET 2024-03-04T16:16:55.349 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:55.349 DEBUG:boto:Data: 2024-03-04T16:16:55.349 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.349 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.349 DEBUG:boto:Params: {} 2024-03-04T16:16:55.349 DEBUG:boto:Token: None 2024-03-04T16:16:55.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.. 2024-03-04T16:16:55.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FonIW4bcBaG1mZWQsT12jlp0fTs= 2024-03-04T16:16:55.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FonIW4bcBaG1mZWQsT12jlp0fTs=', 'Content-Length': '0'} 2024-03-04T16:16:55.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd272484b5047340-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.352 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.352 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.352 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.352 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.352 DEBUG:boto:Method: GET 2024-03-04T16:16:55.352 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:55.352 DEBUG:boto:Data: 2024-03-04T16:16:55.352 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.353 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.353 DEBUG:boto:Params: {} 2024-03-04T16:16:55.353 DEBUG:boto:Token: None 2024-03-04T16:16:55.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031071609cc54327a-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.396 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.397 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.397 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.397 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.397 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:55.397 DEBUG:boto:Method: GET 2024-03-04T16:16:55.397 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:55.397 DEBUG:boto:Data: 2024-03-04T16:16:55.397 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.398 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.398 DEBUG:boto:Params: {} 2024-03-04T16:16:55.398 DEBUG:boto:Token: None 2024-03-04T16:16:55.398 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad71f40072cf88be-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.401 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:55.401 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.401 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.401 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.401 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.401 DEBUG:boto:Method: GET 2024-03-04T16:16:55.401 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.402 DEBUG:boto:Data: 2024-03-04T16:16:55.402 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.402 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.402 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.402 DEBUG:boto:Params: {} 2024-03-04T16:16:55.402 DEBUG:boto:Token: None 2024-03-04T16:16:55.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE= 2024-03-04T16:16:55.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE=', 'Content-Length': '0'} 2024-03-04T16:16:55.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c9198f7c35f8bd4-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.404 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.405 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.405 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.405 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.405 DEBUG:boto:Method: GET 2024-03-04T16:16:55.405 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.405 DEBUG:boto:Data: 2024-03-04T16:16:55.405 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.405 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.405 DEBUG:boto:Params: {} 2024-03-04T16:16:55.405 DEBUG:boto:Token: None 2024-03-04T16:16:55.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE= 2024-03-04T16:16:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE=', 'Content-Length': '0'} 2024-03-04T16:16:55.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045054dd72a5ed28a-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.408 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.408 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.408 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.408 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.408 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.408 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.408 DEBUG:boto:Data: 2024-03-04T16:16:55.409 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.409 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.409 DEBUG:boto:Params: {} 2024-03-04T16:16:55.409 DEBUG:boto:Token: None 2024-03-04T16:16:55.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B9mfWtK2M9A9Lujp5TZIqNv7wW4= 2024-03-04T16:16:55.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B9mfWtK2M9A9Lujp5TZIqNv7wW4=', 'Content-Length': '0'} 2024-03-04T16:16:55.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076dda7b748e723b6-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.411 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.411 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.412 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.412 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.412 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.412 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.412 DEBUG:boto:Data: 2024-03-04T16:16:55.412 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.412 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.412 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.413 DEBUG:boto:Params: {} 2024-03-04T16:16:55.413 DEBUG:boto:Token: None 2024-03-04T16:16:55.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B9mfWtK2M9A9Lujp5TZIqNv7wW4= 2024-03-04T16:16:55.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B9mfWtK2M9A9Lujp5TZIqNv7wW4=', 'Content-Length': '0'} 2024-03-04T16:16:55.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de904c06aca103ce-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:55.415 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.415 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.415 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.416 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.416 DEBUG:boto:Method: GET 2024-03-04T16:16:55.416 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.416 DEBUG:boto:Data: 2024-03-04T16:16:55.416 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.416 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.416 DEBUG:boto:Params: {} 2024-03-04T16:16:55.416 DEBUG:boto:Token: None 2024-03-04T16:16:55.416 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE= 2024-03-04T16:16:55.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE=', 'Content-Length': '0'} 2024-03-04T16:16:55.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1210a01bbaeff85-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.460 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.460 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.460 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.460 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:55.460 DEBUG:boto:Method: GET 2024-03-04T16:16:55.460 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.461 DEBUG:boto:Data: 2024-03-04T16:16:55.461 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.461 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.461 DEBUG:boto:Params: {} 2024-03-04T16:16:55.461 DEBUG:boto:Token: None 2024-03-04T16:16:55.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:55.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE= 2024-03-04T16:16:55.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T1r41gt8ETgaEBjEkPO/4/NFVaE=', 'Content-Length': '0'} 2024-03-04T16:16:55.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c47b74d5f890a27-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.463 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.464 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.464 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.464 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.464 DEBUG:boto:Method: GET 2024-03-04T16:16:55.464 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:55.464 DEBUG:boto:Data: 2024-03-04T16:16:55.464 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.465 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.465 DEBUG:boto:Params: {} 2024-03-04T16:16:55.465 DEBUG:boto:Token: None 2024-03-04T16:16:55.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41cc5234934ecb3-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.508 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.508 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.509 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.509 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.509 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.509 DEBUG:boto:Method: GET 2024-03-04T16:16:55.509 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:55.509 DEBUG:boto:Data: 2024-03-04T16:16:55.509 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.509 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.509 DEBUG:boto:Params: {} 2024-03-04T16:16:55.509 DEBUG:boto:Token: None 2024-03-04T16:16:55.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b6f63888e57e75-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.512 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.513 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.513 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.513 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.513 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:55.513 DEBUG:boto:Data: 2024-03-04T16:16:55.513 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.513 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.513 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.514 DEBUG:boto:Params: {} 2024-03-04T16:16:55.514 DEBUG:boto:Token: None 2024-03-04T16:16:55.514 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/... 2024-03-04T16:16:55.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okqonv8fTC0lt/HyjMETD5NtgpQ= 2024-03-04T16:16:55.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okqonv8fTC0lt/HyjMETD5NtgpQ=', 'Content-Length': '0'} 2024-03-04T16:16:55.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b161362904d365fc-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.516 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.516 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.516 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.516 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:55.516 DEBUG:boto:Data: 2024-03-04T16:16:55.516 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.517 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.517 DEBUG:boto:Params: {} 2024-03-04T16:16:55.517 DEBUG:boto:Token: None 2024-03-04T16:16:55.517 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/... 2024-03-04T16:16:55.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okqonv8fTC0lt/HyjMETD5NtgpQ= 2024-03-04T16:16:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okqonv8fTC0lt/HyjMETD5NtgpQ=', 'Content-Length': '0'} 2024-03-04T16:16:55.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba2ab5f72b790c31-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:55.520 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.520 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.520 DEBUG:boto:Method: GET 2024-03-04T16:16:55.520 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:55.520 DEBUG:boto:Data: 2024-03-04T16:16:55.520 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.520 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.520 DEBUG:boto:Params: {} 2024-03-04T16:16:55.520 DEBUG:boto:Token: None 2024-03-04T16:16:55.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/... 2024-03-04T16:16:55.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zhYQoA2oTZxE8JwiNjiDY+7QMVY= 2024-03-04T16:16:55.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zhYQoA2oTZxE8JwiNjiDY+7QMVY=', 'Content-Length': '0'} 2024-03-04T16:16:55.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f0a4fccbfd159bf-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.564 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:55.564 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:55.564 DEBUG:boto:Method: GET 2024-03-04T16:16:55.564 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:55.564 DEBUG:boto:Data: 2024-03-04T16:16:55.565 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.565 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.565 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.567 DEBUG:boto:Params: {} 2024-03-04T16:16:55.568 DEBUG:boto:Token: None 2024-03-04T16:16:55.568 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/... 2024-03-04T16:16:55.568 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zhYQoA2oTZxE8JwiNjiDY+7QMVY= 2024-03-04T16:16:55.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zhYQoA2oTZxE8JwiNjiDY+7QMVY=', 'Content-Length': '0'} 2024-03-04T16:16:55.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fbae956316ec6a4-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.570 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.570 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.570 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.570 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.571 DEBUG:boto:Method: GET 2024-03-04T16:16:55.571 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:55.571 DEBUG:boto:Data: 2024-03-04T16:16:55.571 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.571 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.571 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.571 DEBUG:boto:Params: {} 2024-03-04T16:16:55.571 DEBUG:boto:Token: None 2024-03-04T16:16:55.571 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d274662be5f17514-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.575 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.575 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.575 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.575 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.575 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:55.575 DEBUG:boto:Method: GET 2024-03-04T16:16:55.575 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:55.575 DEBUG:boto:Data: 2024-03-04T16:16:55.575 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.576 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.576 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.576 DEBUG:boto:Params: {} 2024-03-04T16:16:55.576 DEBUG:boto:Token: None 2024-03-04T16:16:55.576 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.576 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092524c05e8b36d5d-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.579 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:55.579 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.580 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.580 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.580 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.580 DEBUG:boto:Method: GET 2024-03-04T16:16:55.580 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.580 DEBUG:boto:Data: 2024-03-04T16:16:55.580 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.580 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.580 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.580 DEBUG:boto:Params: {} 2024-03-04T16:16:55.580 DEBUG:boto:Token: None 2024-03-04T16:16:55.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY= 2024-03-04T16:16:55.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY=', 'Content-Length': '0'} 2024-03-04T16:16:55.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536eff8feaddebcd-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.628 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.628 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.629 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.629 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.629 DEBUG:boto:Method: GET 2024-03-04T16:16:55.629 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.629 DEBUG:boto:Data: 2024-03-04T16:16:55.629 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.629 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.629 DEBUG:boto:Params: {} 2024-03-04T16:16:55.629 DEBUG:boto:Token: None 2024-03-04T16:16:55.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY= 2024-03-04T16:16:55.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY=', 'Content-Length': '0'} 2024-03-04T16:16:55.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000845adcf8c1595ca2-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.632 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.632 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.633 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.633 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.633 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.633 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.633 DEBUG:boto:Data: 2024-03-04T16:16:55.633 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.633 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.633 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.633 DEBUG:boto:Params: {} 2024-03-04T16:16:55.633 DEBUG:boto:Token: None 2024-03-04T16:16:55.634 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1rFT0XU+Rh7FcF/KpKlYno6BzDo= 2024-03-04T16:16:55.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1rFT0XU+Rh7FcF/KpKlYno6BzDo=', 'Content-Length': '0'} 2024-03-04T16:16:55.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009299785b68f5fedd-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.636 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.636 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.636 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.636 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.636 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.636 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.636 DEBUG:boto:Data: 2024-03-04T16:16:55.636 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.636 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.637 DEBUG:boto:Params: {} 2024-03-04T16:16:55.637 DEBUG:boto:Token: None 2024-03-04T16:16:55.637 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.637 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1rFT0XU+Rh7FcF/KpKlYno6BzDo= 2024-03-04T16:16:55.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1rFT0XU+Rh7FcF/KpKlYno6BzDo=', 'Content-Length': '0'} 2024-03-04T16:16:55.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1e8f9d925350767-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:55.639 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.639 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.639 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.640 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.640 DEBUG:boto:Method: GET 2024-03-04T16:16:55.640 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.640 DEBUG:boto:Data: 2024-03-04T16:16:55.640 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.640 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.640 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.640 DEBUG:boto:Params: {} 2024-03-04T16:16:55.640 DEBUG:boto:Token: None 2024-03-04T16:16:55.640 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.641 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY= 2024-03-04T16:16:55.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY=', 'Content-Length': '0'} 2024-03-04T16:16:55.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c344b7ec9c1e96b-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.684 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.684 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.684 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.685 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.685 DEBUG:boto:Method: GET 2024-03-04T16:16:55.685 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.685 DEBUG:boto:Data: 2024-03-04T16:16:55.685 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.685 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.685 DEBUG:boto:Params: {} 2024-03-04T16:16:55.685 DEBUG:boto:Token: None 2024-03-04T16:16:55.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:55.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY= 2024-03-04T16:16:55.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bovz4It8yLAd0OOPXY/4ew8MOSY=', 'Content-Length': '0'} 2024-03-04T16:16:55.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1b0ad7fe5ddbc84-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.688 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.688 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.688 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.689 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.689 DEBUG:boto:Method: GET 2024-03-04T16:16:55.689 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:55.689 DEBUG:boto:Data: 2024-03-04T16:16:55.689 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.689 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.689 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.689 DEBUG:boto:Params: {} 2024-03-04T16:16:55.689 DEBUG:boto:Token: None 2024-03-04T16:16:55.689 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d968b46d6ca59fe-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.692 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.693 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.693 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.693 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.693 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.693 DEBUG:boto:Method: GET 2024-03-04T16:16:55.693 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:55.693 DEBUG:boto:Data: 2024-03-04T16:16:55.693 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.694 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.694 DEBUG:boto:Params: {} 2024-03-04T16:16:55.694 DEBUG:boto:Token: None 2024-03-04T16:16:55.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89c1882ddad7d21-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.697 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.697 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.697 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.697 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.697 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:55.698 DEBUG:boto:Data: 2024-03-04T16:16:55.698 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.698 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.698 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.698 DEBUG:boto:Params: {} 2024-03-04T16:16:55.698 DEBUG:boto:Token: None 2024-03-04T16:16:55.698 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o 2024-03-04T16:16:55.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F9776zlAM+1EyUFX2W9v5F3Hueg= 2024-03-04T16:16:55.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F9776zlAM+1EyUFX2W9v5F3Hueg=', 'Content-Length': '0'} 2024-03-04T16:16:55.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e9f2f4b6f5ae1a-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.700 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.700 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.700 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.700 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:55.701 DEBUG:boto:Data: 2024-03-04T16:16:55.701 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.701 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.701 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.701 DEBUG:boto:Params: {} 2024-03-04T16:16:55.701 DEBUG:boto:Token: None 2024-03-04T16:16:55.701 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o 2024-03-04T16:16:55.701 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F9776zlAM+1EyUFX2W9v5F3Hueg= 2024-03-04T16:16:55.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F9776zlAM+1EyUFX2W9v5F3Hueg=', 'Content-Length': '0'} 2024-03-04T16:16:55.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d890887a0fa14fe4-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:55.703 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.703 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.703 DEBUG:boto:Method: GET 2024-03-04T16:16:55.704 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:55.704 DEBUG:boto:Data: 2024-03-04T16:16:55.704 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.704 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.704 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.704 DEBUG:boto:Params: {} 2024-03-04T16:16:55.704 DEBUG:boto:Token: None 2024-03-04T16:16:55.704 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o 2024-03-04T16:16:55.704 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V14zm9ZYsREjcLPUaa+LTpmTzLw= 2024-03-04T16:16:55.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V14zm9ZYsREjcLPUaa+LTpmTzLw=', 'Content-Length': '0'} 2024-03-04T16:16:55.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000607025bd43bc8c7-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.748 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:55.748 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:55.748 DEBUG:boto:Method: GET 2024-03-04T16:16:55.748 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:55.748 DEBUG:boto:Data: 2024-03-04T16:16:55.748 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.749 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.749 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.749 DEBUG:boto:Params: {} 2024-03-04T16:16:55.749 DEBUG:boto:Token: None 2024-03-04T16:16:55.749 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o 2024-03-04T16:16:55.749 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V14zm9ZYsREjcLPUaa+LTpmTzLw= 2024-03-04T16:16:55.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V14zm9ZYsREjcLPUaa+LTpmTzLw=', 'Content-Length': '0'} 2024-03-04T16:16:55.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075942533cbde1de1-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.751 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.751 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.751 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.751 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.752 DEBUG:boto:Method: GET 2024-03-04T16:16:55.752 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:55.752 DEBUG:boto:Data: 2024-03-04T16:16:55.752 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.752 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.752 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.752 DEBUG:boto:Params: {} 2024-03-04T16:16:55.752 DEBUG:boto:Token: None 2024-03-04T16:16:55.752 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.752 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e17298aecf452753-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.796 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.796 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.797 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.797 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.797 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:55.797 DEBUG:boto:Method: GET 2024-03-04T16:16:55.797 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:55.797 DEBUG:boto:Data: 2024-03-04T16:16:55.797 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.797 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.797 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.797 DEBUG:boto:Params: {} 2024-03-04T16:16:55.797 DEBUG:boto:Token: None 2024-03-04T16:16:55.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf0e08076856433-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.801 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:55.801 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.801 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.801 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.801 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.801 DEBUG:boto:Method: GET 2024-03-04T16:16:55.802 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.802 DEBUG:boto:Data: 2024-03-04T16:16:55.802 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.802 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.802 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.802 DEBUG:boto:Params: {} 2024-03-04T16:16:55.802 DEBUG:boto:Token: None 2024-03-04T16:16:55.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI= 2024-03-04T16:16:55.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI=', 'Content-Length': '0'} 2024-03-04T16:16:55.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea3642d177e4476f-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.805 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.805 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.805 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.805 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.805 DEBUG:boto:Method: GET 2024-03-04T16:16:55.805 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.805 DEBUG:boto:Data: 2024-03-04T16:16:55.805 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.805 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.805 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.805 DEBUG:boto:Params: {} 2024-03-04T16:16:55.805 DEBUG:boto:Token: None 2024-03-04T16:16:55.806 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.806 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI= 2024-03-04T16:16:55.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI=', 'Content-Length': '0'} 2024-03-04T16:16:55.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dace1c1cb81513b-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.808 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.808 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.808 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.808 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.808 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.808 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.809 DEBUG:boto:Data: 2024-03-04T16:16:55.809 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.809 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.809 DEBUG:boto:Params: {} 2024-03-04T16:16:55.809 DEBUG:boto:Token: None 2024-03-04T16:16:55.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DA97uZeQNR0DpxaJMa+FFcPKFC4= 2024-03-04T16:16:55.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DA97uZeQNR0DpxaJMa+FFcPKFC4=', 'Content-Length': '0'} 2024-03-04T16:16:55.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061e6c93fe5b35784-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.811 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.812 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.812 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.812 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.812 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.812 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.812 DEBUG:boto:Data: 2024-03-04T16:16:55.812 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.812 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.812 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.812 DEBUG:boto:Params: {} 2024-03-04T16:16:55.812 DEBUG:boto:Token: None 2024-03-04T16:16:55.813 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.813 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DA97uZeQNR0DpxaJMa+FFcPKFC4= 2024-03-04T16:16:55.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DA97uZeQNR0DpxaJMa+FFcPKFC4=', 'Content-Length': '0'} 2024-03-04T16:16:55.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073b4136697fcce0d-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:55.815 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.815 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.815 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.815 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.815 DEBUG:boto:Method: GET 2024-03-04T16:16:55.815 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.816 DEBUG:boto:Data: 2024-03-04T16:16:55.816 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.816 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.816 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.816 DEBUG:boto:Params: {} 2024-03-04T16:16:55.816 DEBUG:boto:Token: None 2024-03-04T16:16:55.816 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.816 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI= 2024-03-04T16:16:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI=', 'Content-Length': '0'} 2024-03-04T16:16:55.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0737d5b7a8a7f12-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.860 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.860 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.860 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.860 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.860 DEBUG:boto:Method: GET 2024-03-04T16:16:55.861 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.861 DEBUG:boto:Data: 2024-03-04T16:16:55.861 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.861 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.861 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.861 DEBUG:boto:Params: {} 2024-03-04T16:16:55.861 DEBUG:boto:Token: None 2024-03-04T16:16:55.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:55.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI= 2024-03-04T16:16:55.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NngHrlJQvyOmWlggyOf995HXAiI=', 'Content-Length': '0'} 2024-03-04T16:16:55.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002145c232508a5de0-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.863 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.863 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.864 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.864 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.864 DEBUG:boto:Method: GET 2024-03-04T16:16:55.864 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.864 DEBUG:boto:Data: 2024-03-04T16:16:55.864 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.864 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.864 DEBUG:boto:Params: {} 2024-03-04T16:16:55.864 DEBUG:boto:Token: None 2024-03-04T16:16:55.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0a8803d2a726af-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.908 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.908 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.909 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.909 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.909 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.909 DEBUG:boto:Method: GET 2024-03-04T16:16:55.909 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.909 DEBUG:boto:Data: 2024-03-04T16:16:55.909 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.909 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.909 DEBUG:boto:Params: {} 2024-03-04T16:16:55.909 DEBUG:boto:Token: None 2024-03-04T16:16:55.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014552f76f7e9f3e7-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.912 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:55.913 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.913 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.913 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.913 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:55.913 DEBUG:boto:Data: 2024-03-04T16:16:55.913 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.913 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.913 DEBUG:boto:Params: {} 2024-03-04T16:16:55.914 DEBUG:boto:Token: None 2024-03-04T16:16:55.914 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o. 2024-03-04T16:16:55.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wI3YcFTwNx2sluJ0vNf8OtaxHVA= 2024-03-04T16:16:55.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wI3YcFTwNx2sluJ0vNf8OtaxHVA=', 'Content-Length': '0'} 2024-03-04T16:16:55.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d11e6b0b0394f92b-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.916 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.916 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.917 DEBUG:boto:Method: HEAD 2024-03-04T16:16:55.917 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:55.917 DEBUG:boto:Data: 2024-03-04T16:16:55.917 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.917 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.917 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.917 DEBUG:boto:Params: {} 2024-03-04T16:16:55.917 DEBUG:boto:Token: None 2024-03-04T16:16:55.917 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o. 2024-03-04T16:16:55.918 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wI3YcFTwNx2sluJ0vNf8OtaxHVA= 2024-03-04T16:16:55.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wI3YcFTwNx2sluJ0vNf8OtaxHVA=', 'Content-Length': '0'} 2024-03-04T16:16:55.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f145c8c07401ea6-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:55.920 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.920 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.920 DEBUG:boto:Method: GET 2024-03-04T16:16:55.920 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:55.920 DEBUG:boto:Data: 2024-03-04T16:16:55.920 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.920 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.920 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.920 DEBUG:boto:Params: {} 2024-03-04T16:16:55.921 DEBUG:boto:Token: None 2024-03-04T16:16:55.921 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o. 2024-03-04T16:16:55.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jFJHqE1lCfO3BhetHgSvMNN/TEw= 2024-03-04T16:16:55.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jFJHqE1lCfO3BhetHgSvMNN/TEw=', 'Content-Length': '0'} 2024-03-04T16:16:55.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089dc8533b1ea3456-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.964 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:55.964 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:55.964 DEBUG:boto:Method: GET 2024-03-04T16:16:55.964 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:55.965 DEBUG:boto:Data: 2024-03-04T16:16:55.965 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.965 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:55.965 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.965 DEBUG:boto:Params: {} 2024-03-04T16:16:55.965 DEBUG:boto:Token: None 2024-03-04T16:16:55.965 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/.o. 2024-03-04T16:16:55.965 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jFJHqE1lCfO3BhetHgSvMNN/TEw= 2024-03-04T16:16:55.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jFJHqE1lCfO3BhetHgSvMNN/TEw=', 'Content-Length': '0'} 2024-03-04T16:16:55.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009537ba9a269b231d-0065e5f3f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:55.967 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:55.967 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:55.968 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.968 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.968 DEBUG:boto:Method: GET 2024-03-04T16:16:55.968 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:55.968 DEBUG:boto:Data: 2024-03-04T16:16:55.968 DEBUG:boto:Headers: {} 2024-03-04T16:16:55.968 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:55.968 DEBUG:boto:Port: 8000 2024-03-04T16:16:55.968 DEBUG:boto:Params: {} 2024-03-04T16:16:55.968 DEBUG:boto:Token: None 2024-03-04T16:16:55.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:55 GMT /sbnvju-11/ 2024-03-04T16:16:55.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM= 2024-03-04T16:16:55.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qC2yKWLByDueh2EGNqzcHxAhpmM=', 'Content-Length': '0'} 2024-03-04T16:16:55.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bb330a9f3dd5ab-0065e5f3f7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.012 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.012 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.012 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.013 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:56.013 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:56.013 DEBUG:boto:Method: GET 2024-03-04T16:16:56.013 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:56.013 DEBUG:boto:Data: 2024-03-04T16:16:56.013 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.013 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.013 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.013 DEBUG:boto:Params: {} 2024-03-04T16:16:56.013 DEBUG:boto:Token: None 2024-03-04T16:16:56.014 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e32ed346a1c27f4-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.016 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:56.017 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.017 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.017 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.017 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.017 DEBUG:boto:Method: GET 2024-03-04T16:16:56.017 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.017 DEBUG:boto:Data: 2024-03-04T16:16:56.018 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.018 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.018 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.018 DEBUG:boto:Params: {} 2024-03-04T16:16:56.018 DEBUG:boto:Token: None 2024-03-04T16:16:56.018 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.018 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No= 2024-03-04T16:16:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No=', 'Content-Length': '0'} 2024-03-04T16:16:56.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d1b86d880223f1-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.064 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.064 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.064 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.064 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.064 DEBUG:boto:Method: GET 2024-03-04T16:16:56.064 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.065 DEBUG:boto:Data: 2024-03-04T16:16:56.065 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.065 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.065 DEBUG:boto:Params: {} 2024-03-04T16:16:56.065 DEBUG:boto:Token: None 2024-03-04T16:16:56.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No= 2024-03-04T16:16:56.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No=', 'Content-Length': '0'} 2024-03-04T16:16:56.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f668c91059315f28-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.067 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.068 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.068 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.068 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.068 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.068 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.068 DEBUG:boto:Data: 2024-03-04T16:16:56.068 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.068 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.068 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.068 DEBUG:boto:Params: {} 2024-03-04T16:16:56.068 DEBUG:boto:Token: None 2024-03-04T16:16:56.069 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8tiibPt4Y6qp/G4CvYwGDhuWxtk= 2024-03-04T16:16:56.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8tiibPt4Y6qp/G4CvYwGDhuWxtk=', 'Content-Length': '0'} 2024-03-04T16:16:56.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003978b1ce5fe0cf70-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.071 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.071 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.071 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.071 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.071 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.071 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.071 DEBUG:boto:Data: 2024-03-04T16:16:56.071 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.072 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.072 DEBUG:boto:Params: {} 2024-03-04T16:16:56.072 DEBUG:boto:Token: None 2024-03-04T16:16:56.072 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.072 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8tiibPt4Y6qp/G4CvYwGDhuWxtk= 2024-03-04T16:16:56.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8tiibPt4Y6qp/G4CvYwGDhuWxtk=', 'Content-Length': '0'} 2024-03-04T16:16:56.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033c0a5040a217789-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:56.075 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.075 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.075 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.075 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.075 DEBUG:boto:Method: GET 2024-03-04T16:16:56.075 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.076 DEBUG:boto:Data: 2024-03-04T16:16:56.076 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.076 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.076 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.076 DEBUG:boto:Params: {} 2024-03-04T16:16:56.076 DEBUG:boto:Token: None 2024-03-04T16:16:56.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No= 2024-03-04T16:16:56.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No=', 'Content-Length': '0'} 2024-03-04T16:16:56.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000645c30322f5133a3-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.120 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.120 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.120 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.120 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.120 DEBUG:boto:Method: GET 2024-03-04T16:16:56.121 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.121 DEBUG:boto:Data: 2024-03-04T16:16:56.121 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.121 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.121 DEBUG:boto:Params: {} 2024-03-04T16:16:56.121 DEBUG:boto:Token: None 2024-03-04T16:16:56.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:56.121 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No= 2024-03-04T16:16:56.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xtqcdd/k0ABGrxDWFuGeCXpf1No=', 'Content-Length': '0'} 2024-03-04T16:16:56.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6274782a11c256-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.124 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.124 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.124 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.124 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.124 DEBUG:boto:Method: GET 2024-03-04T16:16:56.124 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.124 DEBUG:boto:Data: 2024-03-04T16:16:56.124 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.124 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.124 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.125 DEBUG:boto:Params: {} 2024-03-04T16:16:56.125 DEBUG:boto:Token: None 2024-03-04T16:16:56.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a75633eb7b95098-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.168 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.168 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.169 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.169 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.169 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.169 DEBUG:boto:Method: GET 2024-03-04T16:16:56.169 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.169 DEBUG:boto:Data: 2024-03-04T16:16:56.169 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.169 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.169 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.169 DEBUG:boto:Params: {} 2024-03-04T16:16:56.170 DEBUG:boto:Token: None 2024-03-04T16:16:56.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ecf68d90a9d0338-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.172 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.173 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.173 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.173 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.173 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:56.173 DEBUG:boto:Data: 2024-03-04T16:16:56.173 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.174 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.174 DEBUG:boto:Params: {} 2024-03-04T16:16:56.174 DEBUG:boto:Token: None 2024-03-04T16:16:56.174 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A 2024-03-04T16:16:56.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:prg3Zaq9mNHh111ygyPQeJfi7gs= 2024-03-04T16:16:56.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:prg3Zaq9mNHh111ygyPQeJfi7gs=', 'Content-Length': '0'} 2024-03-04T16:16:56.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cc0f3ea2c98f76f-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.177 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.177 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.177 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.177 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:56.177 DEBUG:boto:Data: 2024-03-04T16:16:56.177 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.177 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.177 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.177 DEBUG:boto:Params: {} 2024-03-04T16:16:56.177 DEBUG:boto:Token: None 2024-03-04T16:16:56.177 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A 2024-03-04T16:16:56.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:prg3Zaq9mNHh111ygyPQeJfi7gs= 2024-03-04T16:16:56.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:prg3Zaq9mNHh111ygyPQeJfi7gs=', 'Content-Length': '0'} 2024-03-04T16:16:56.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bba513e436e87b37-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:56.180 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.180 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.180 DEBUG:boto:Method: GET 2024-03-04T16:16:56.180 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:56.181 DEBUG:boto:Data: 2024-03-04T16:16:56.181 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.181 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.181 DEBUG:boto:Params: {} 2024-03-04T16:16:56.181 DEBUG:boto:Token: None 2024-03-04T16:16:56.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A 2024-03-04T16:16:56.181 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Sa4ASLhqezz8e2G05T698zyBfAA= 2024-03-04T16:16:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Sa4ASLhqezz8e2G05T698zyBfAA=', 'Content-Length': '0'} 2024-03-04T16:16:56.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fdc42d9ebe14358-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.224 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:56.224 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:56.224 DEBUG:boto:Method: GET 2024-03-04T16:16:56.224 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:56.225 DEBUG:boto:Data: 2024-03-04T16:16:56.225 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.225 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.225 DEBUG:boto:Params: {} 2024-03-04T16:16:56.225 DEBUG:boto:Token: None 2024-03-04T16:16:56.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%3A 2024-03-04T16:16:56.225 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Sa4ASLhqezz8e2G05T698zyBfAA= 2024-03-04T16:16:56.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Sa4ASLhqezz8e2G05T698zyBfAA=', 'Content-Length': '0'} 2024-03-04T16:16:56.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f704f79af23e02b-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.227 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.228 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.228 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.228 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.228 DEBUG:boto:Method: GET 2024-03-04T16:16:56.228 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.228 DEBUG:boto:Data: 2024-03-04T16:16:56.228 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.228 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.228 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.228 DEBUG:boto:Params: {} 2024-03-04T16:16:56.228 DEBUG:boto:Token: None 2024-03-04T16:16:56.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060de4d5dc1dae12-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.272 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.273 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.273 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.273 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.273 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.273 DEBUG:boto:Method: GET 2024-03-04T16:16:56.274 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:56.274 DEBUG:boto:Data: 2024-03-04T16:16:56.274 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.274 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.274 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.274 DEBUG:boto:Params: {} 2024-03-04T16:16:56.275 DEBUG:boto:Token: None 2024-03-04T16:16:56.275 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351d78b56f4a7ebe-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.278 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:56.279 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.279 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.279 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.279 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.279 DEBUG:boto:Method: GET 2024-03-04T16:16:56.279 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.279 DEBUG:boto:Data: 2024-03-04T16:16:56.279 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.279 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.279 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.279 DEBUG:boto:Params: {} 2024-03-04T16:16:56.279 DEBUG:boto:Token: None 2024-03-04T16:16:56.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E= 2024-03-04T16:16:56.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E=', 'Content-Length': '0'} 2024-03-04T16:16:56.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064d7e09d9e5935b5-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.328 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.328 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.328 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.328 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.328 DEBUG:boto:Method: GET 2024-03-04T16:16:56.329 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.329 DEBUG:boto:Data: 2024-03-04T16:16:56.329 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.329 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.329 DEBUG:boto:Params: {} 2024-03-04T16:16:56.329 DEBUG:boto:Token: None 2024-03-04T16:16:56.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E= 2024-03-04T16:16:56.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E=', 'Content-Length': '0'} 2024-03-04T16:16:56.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff419d53dc7d4e3e-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.332 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.332 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.332 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.332 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.332 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.332 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.332 DEBUG:boto:Data: 2024-03-04T16:16:56.333 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.333 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.333 DEBUG:boto:Params: {} 2024-03-04T16:16:56.333 DEBUG:boto:Token: None 2024-03-04T16:16:56.333 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uJjPHkewAll+9dV60SVTRvYvAhE= 2024-03-04T16:16:56.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uJjPHkewAll+9dV60SVTRvYvAhE=', 'Content-Length': '0'} 2024-03-04T16:16:56.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dce6dd237de889d6-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.336 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.336 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.336 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.336 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.336 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.336 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.336 DEBUG:boto:Data: 2024-03-04T16:16:56.337 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.337 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.337 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.337 DEBUG:boto:Params: {} 2024-03-04T16:16:56.337 DEBUG:boto:Token: None 2024-03-04T16:16:56.338 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uJjPHkewAll+9dV60SVTRvYvAhE= 2024-03-04T16:16:56.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uJjPHkewAll+9dV60SVTRvYvAhE=', 'Content-Length': '0'} 2024-03-04T16:16:56.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef77302a361d31e7-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:56.340 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.340 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.341 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.341 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.341 DEBUG:boto:Method: GET 2024-03-04T16:16:56.341 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.341 DEBUG:boto:Data: 2024-03-04T16:16:56.341 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.341 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.341 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.341 DEBUG:boto:Params: {} 2024-03-04T16:16:56.341 DEBUG:boto:Token: None 2024-03-04T16:16:56.341 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E= 2024-03-04T16:16:56.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E=', 'Content-Length': '0'} 2024-03-04T16:16:56.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccfc0ea8bc5fd8f4-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.388 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.388 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.388 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.388 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.388 DEBUG:boto:Method: GET 2024-03-04T16:16:56.389 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.389 DEBUG:boto:Data: 2024-03-04T16:16:56.389 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.389 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.389 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.389 DEBUG:boto:Params: {} 2024-03-04T16:16:56.389 DEBUG:boto:Token: None 2024-03-04T16:16:56.389 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:56.389 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E= 2024-03-04T16:16:56.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZzSq47HGcMu9QMXmV4Oum5Hz+4E=', 'Content-Length': '0'} 2024-03-04T16:16:56.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c60d912c3138d65b-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.392 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.392 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.392 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.392 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.392 DEBUG:boto:Method: GET 2024-03-04T16:16:56.392 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.393 DEBUG:boto:Data: 2024-03-04T16:16:56.393 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.393 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.393 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.393 DEBUG:boto:Params: {} 2024-03-04T16:16:56.393 DEBUG:boto:Token: None 2024-03-04T16:16:56.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.393 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e9749bf82a4ed3-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.436 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.436 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.437 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.437 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.437 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.437 DEBUG:boto:Method: GET 2024-03-04T16:16:56.437 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.437 DEBUG:boto:Data: 2024-03-04T16:16:56.437 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.437 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.437 DEBUG:boto:Params: {} 2024-03-04T16:16:56.437 DEBUG:boto:Token: None 2024-03-04T16:16:56.438 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.438 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000346f0ce7fbb52c24-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.440 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.441 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.441 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.441 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.441 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:56.441 DEBUG:boto:Data: 2024-03-04T16:16:56.441 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.441 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.441 DEBUG:boto:Params: {} 2024-03-04T16:16:56.441 DEBUG:boto:Token: None 2024-03-04T16:16:56.442 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj 2024-03-04T16:16:56.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p9cmBZDR6Cn0fdWnRh13u8wV0jI= 2024-03-04T16:16:56.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p9cmBZDR6Cn0fdWnRh13u8wV0jI=', 'Content-Length': '0'} 2024-03-04T16:16:56.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051fdde9e1c4e8e5e-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.444 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.444 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.444 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.444 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:56.444 DEBUG:boto:Data: 2024-03-04T16:16:56.444 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.444 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.444 DEBUG:boto:Params: {} 2024-03-04T16:16:56.444 DEBUG:boto:Token: None 2024-03-04T16:16:56.445 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj 2024-03-04T16:16:56.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p9cmBZDR6Cn0fdWnRh13u8wV0jI= 2024-03-04T16:16:56.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p9cmBZDR6Cn0fdWnRh13u8wV0jI=', 'Content-Length': '0'} 2024-03-04T16:16:56.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a17e72d6e353a857-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:56.447 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.447 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.447 DEBUG:boto:Method: GET 2024-03-04T16:16:56.447 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:56.447 DEBUG:boto:Data: 2024-03-04T16:16:56.447 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.447 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.447 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.447 DEBUG:boto:Params: {} 2024-03-04T16:16:56.448 DEBUG:boto:Token: None 2024-03-04T16:16:56.448 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj 2024-03-04T16:16:56.448 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:00UKyyMD3029IBlXDAPnS+h+RLo= 2024-03-04T16:16:56.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:00UKyyMD3029IBlXDAPnS+h+RLo=', 'Content-Length': '0'} 2024-03-04T16:16:56.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e705d46cc36ac65-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.492 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:56.492 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:56.492 DEBUG:boto:Method: GET 2024-03-04T16:16:56.492 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:56.493 DEBUG:boto:Data: 2024-03-04T16:16:56.493 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.493 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.493 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.493 DEBUG:boto:Params: {} 2024-03-04T16:16:56.493 DEBUG:boto:Token: None 2024-03-04T16:16:56.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/_myobj 2024-03-04T16:16:56.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:00UKyyMD3029IBlXDAPnS+h+RLo= 2024-03-04T16:16:56.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:00UKyyMD3029IBlXDAPnS+h+RLo=', 'Content-Length': '0'} 2024-03-04T16:16:56.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067d8a24ddfae45a9-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.495 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.496 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.496 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.496 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.496 DEBUG:boto:Method: GET 2024-03-04T16:16:56.496 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.496 DEBUG:boto:Data: 2024-03-04T16:16:56.496 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.496 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.496 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.496 DEBUG:boto:Params: {} 2024-03-04T16:16:56.496 DEBUG:boto:Token: None 2024-03-04T16:16:56.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d49377f16904aad8-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.499 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.500 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.500 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.500 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.500 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.500 DEBUG:boto:Method: GET 2024-03-04T16:16:56.500 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:56.500 DEBUG:boto:Data: 2024-03-04T16:16:56.500 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.500 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.500 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.500 DEBUG:boto:Params: {} 2024-03-04T16:16:56.501 DEBUG:boto:Token: None 2024-03-04T16:16:56.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c696087dadba08f6-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.503 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:56.504 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.504 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.504 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.504 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.504 DEBUG:boto:Method: GET 2024-03-04T16:16:56.504 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.504 DEBUG:boto:Data: 2024-03-04T16:16:56.504 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.504 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.505 DEBUG:boto:Params: {} 2024-03-04T16:16:56.505 DEBUG:boto:Token: None 2024-03-04T16:16:56.505 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.505 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM= 2024-03-04T16:16:56.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM=', 'Content-Length': '0'} 2024-03-04T16:16:56.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc7b04f4af1c4077-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.507 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.507 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.507 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.507 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.507 DEBUG:boto:Method: GET 2024-03-04T16:16:56.507 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.508 DEBUG:boto:Data: 2024-03-04T16:16:56.508 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.508 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.508 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.508 DEBUG:boto:Params: {} 2024-03-04T16:16:56.508 DEBUG:boto:Token: None 2024-03-04T16:16:56.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM= 2024-03-04T16:16:56.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM=', 'Content-Length': '0'} 2024-03-04T16:16:56.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f3214c2e51a8d7c-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.511 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.511 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.511 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.511 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.511 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.511 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.511 DEBUG:boto:Data: 2024-03-04T16:16:56.511 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.511 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.511 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.511 DEBUG:boto:Params: {} 2024-03-04T16:16:56.512 DEBUG:boto:Token: None 2024-03-04T16:16:56.512 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.512 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KD4k0jtv/1Kc8NvVivhItd1d2co= 2024-03-04T16:16:56.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KD4k0jtv/1Kc8NvVivhItd1d2co=', 'Content-Length': '0'} 2024-03-04T16:16:56.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069a0a485bca193ad-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.514 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.514 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.515 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.515 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.515 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.515 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.515 DEBUG:boto:Data: 2024-03-04T16:16:56.515 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.515 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.515 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.515 DEBUG:boto:Params: {} 2024-03-04T16:16:56.515 DEBUG:boto:Token: None 2024-03-04T16:16:56.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KD4k0jtv/1Kc8NvVivhItd1d2co= 2024-03-04T16:16:56.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KD4k0jtv/1Kc8NvVivhItd1d2co=', 'Content-Length': '0'} 2024-03-04T16:16:56.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cdc836fcf020648-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:56.518 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.518 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.518 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.518 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.518 DEBUG:boto:Method: GET 2024-03-04T16:16:56.518 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.518 DEBUG:boto:Data: 2024-03-04T16:16:56.518 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.518 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.519 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.519 DEBUG:boto:Params: {} 2024-03-04T16:16:56.519 DEBUG:boto:Token: None 2024-03-04T16:16:56.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM= 2024-03-04T16:16:56.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM=', 'Content-Length': '0'} 2024-03-04T16:16:56.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a659f858a019cae7-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.564 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.564 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.564 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.564 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.564 DEBUG:boto:Method: GET 2024-03-04T16:16:56.564 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.565 DEBUG:boto:Data: 2024-03-04T16:16:56.565 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.565 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.565 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.565 DEBUG:boto:Params: {} 2024-03-04T16:16:56.565 DEBUG:boto:Token: None 2024-03-04T16:16:56.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:56.565 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM= 2024-03-04T16:16:56.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rzm47rPJWebNpbd3o72pUNt32AM=', 'Content-Length': '0'} 2024-03-04T16:16:56.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9531ee0bad72d63-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.567 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.568 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.568 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.568 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.568 DEBUG:boto:Method: GET 2024-03-04T16:16:56.568 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.568 DEBUG:boto:Data: 2024-03-04T16:16:56.568 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.568 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.568 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.568 DEBUG:boto:Params: {} 2024-03-04T16:16:56.569 DEBUG:boto:Token: None 2024-03-04T16:16:56.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.569 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af523266bd8fc626-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.616 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.616 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.616 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.617 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.617 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.617 DEBUG:boto:Method: GET 2024-03-04T16:16:56.617 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.617 DEBUG:boto:Data: 2024-03-04T16:16:56.617 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.617 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.617 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.617 DEBUG:boto:Params: {} 2024-03-04T16:16:56.617 DEBUG:boto:Token: None 2024-03-04T16:16:56.617 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074bb455d12a263dc-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.620 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.621 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.621 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.621 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.621 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:56.621 DEBUG:boto:Data: 2024-03-04T16:16:56.621 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.621 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.621 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.621 DEBUG:boto:Params: {} 2024-03-04T16:16:56.621 DEBUG:boto:Token: None 2024-03-04T16:16:56.622 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj 2024-03-04T16:16:56.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CS3ZVep+4wM3DUNv/pqeAPBSs7g= 2024-03-04T16:16:56.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CS3ZVep+4wM3DUNv/pqeAPBSs7g=', 'Content-Length': '0'} 2024-03-04T16:16:56.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097d37ae61b079fc2-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.624 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.624 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.624 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.624 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:56.624 DEBUG:boto:Data: 2024-03-04T16:16:56.624 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.624 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.624 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.624 DEBUG:boto:Params: {} 2024-03-04T16:16:56.625 DEBUG:boto:Token: None 2024-03-04T16:16:56.625 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj 2024-03-04T16:16:56.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CS3ZVep+4wM3DUNv/pqeAPBSs7g= 2024-03-04T16:16:56.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CS3ZVep+4wM3DUNv/pqeAPBSs7g=', 'Content-Length': '0'} 2024-03-04T16:16:56.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3d6dc896ef8e1e7-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:56.627 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.627 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.627 DEBUG:boto:Method: GET 2024-03-04T16:16:56.627 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:56.627 DEBUG:boto:Data: 2024-03-04T16:16:56.627 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.628 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.628 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.628 DEBUG:boto:Params: {} 2024-03-04T16:16:56.628 DEBUG:boto:Token: None 2024-03-04T16:16:56.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj 2024-03-04T16:16:56.628 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wB8FGuHPfF/sNf+Vy4lrqqHirx4= 2024-03-04T16:16:56.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wB8FGuHPfF/sNf+Vy4lrqqHirx4=', 'Content-Length': '0'} 2024-03-04T16:16:56.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af75b09c7167e3e7-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.672 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:56.672 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:56.672 DEBUG:boto:Method: GET 2024-03-04T16:16:56.672 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:56.672 DEBUG:boto:Data: 2024-03-04T16:16:56.672 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.672 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.673 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.673 DEBUG:boto:Params: {} 2024-03-04T16:16:56.673 DEBUG:boto:Token: None 2024-03-04T16:16:56.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/myobj 2024-03-04T16:16:56.673 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wB8FGuHPfF/sNf+Vy4lrqqHirx4= 2024-03-04T16:16:56.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wB8FGuHPfF/sNf+Vy4lrqqHirx4=', 'Content-Length': '0'} 2024-03-04T16:16:56.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d33ff3d04a4394f2-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.675 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.675 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.675 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.675 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.675 DEBUG:boto:Method: GET 2024-03-04T16:16:56.676 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.676 DEBUG:boto:Data: 2024-03-04T16:16:56.676 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.676 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:56.676 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.676 DEBUG:boto:Params: {} 2024-03-04T16:16:56.676 DEBUG:boto:Token: None 2024-03-04T16:16:56.676 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.676 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff3ebcb062b4b478-0065e5f3f8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.679 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.680 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.680 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.680 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.680 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.680 DEBUG:boto:Method: GET 2024-03-04T16:16:56.680 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:56.680 DEBUG:boto:Data: 2024-03-04T16:16:56.680 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.680 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.681 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.681 DEBUG:boto:Params: {} 2024-03-04T16:16:56.681 DEBUG:boto:Token: None 2024-03-04T16:16:56.681 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.681 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e66cf4ec5cd689c-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.684 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.684 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-11 zones={test-zone1, test-zone2} 2024-03-04T16:16:56.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:16:56.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:16:56.685 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:16:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:16:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:16:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:56.804+0000 7f4837412800 1 bucket sync caught up with source: 2024-03-04T16:16:56.805 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.21.6, , 00000000001.18.6, , 00000000001.13.6, 00000000002.15.6, 00000000001.7.6, 00000000001.5.6, 00000000001.11.6, ] 2024-03-04T16:16:56.805 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.21.6, , 00000000001.18.6, , 00000000001.13.6, 00000000002.15.6, 00000000001.7.6, 00000000001.5.6, 00000000001.11.6, ] 2024-03-04T16:16:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:56.804+0000 7f4837412800 0 bucket checkpoint complete 2024-03-04T16:16:56.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-11 zones={test-zone3, test-zone2} 2024-03-04T16:16:56.825 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.825 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.825 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.826 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:16:56.826 DEBUG:boto:Data: 2024-03-04T16:16:56.826 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.826 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.826 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.827 DEBUG:boto:Params: {} 2024-03-04T16:16:56.827 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:16:56.827 DEBUG:boto:Token: None 2024-03-04T16:16:56.827 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.828 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dSRGHKCRF/gth2M6M2odni7fugo= 2024-03-04T16:16:56.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dSRGHKCRF/gth2M6M2odni7fugo=', 'Content-Length': '0'} 2024-03-04T16:16:56.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004da594cf2d044713-0065e5f3f8-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.833 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.833 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.833 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.833 DEBUG:boto:Path: /sbnvju-11/ 2024-03-04T16:16:56.834 DEBUG:boto:Data: 2024-03-04T16:16:56.834 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.834 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.834 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.834 DEBUG:boto:Params: {} 2024-03-04T16:16:56.835 DEBUG:boto:Token: None 2024-03-04T16:16:56.835 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.835 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dSRGHKCRF/gth2M6M2odni7fugo= 2024-03-04T16:16:56.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dSRGHKCRF/gth2M6M2odni7fugo=', 'Content-Length': '0'} 2024-03-04T16:16:56.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031507b6bf06ca719-0065e5f3f8-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:16:56.839 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.839 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.839 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.839 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.839 DEBUG:boto:Method: GET 2024-03-04T16:16:56.840 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.840 DEBUG:boto:Data: 2024-03-04T16:16:56.840 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.840 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.840 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.841 DEBUG:boto:Params: {} 2024-03-04T16:16:56.841 DEBUG:boto:Token: None 2024-03-04T16:16:56.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/?versions 2024-03-04T16:16:56.841 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA= 2024-03-04T16:16:56.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA=', 'Content-Length': '0'} 2024-03-04T16:16:56.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fef6eab60d8748a-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.844 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:56.845 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:56.846 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:56.846 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:56.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:16:56.846 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.846 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.846 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.846 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.846 DEBUG:boto:Method: GET 2024-03-04T16:16:56.846 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.847 DEBUG:boto:Data: 2024-03-04T16:16:56.847 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.847 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.847 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.847 DEBUG:boto:Params: {} 2024-03-04T16:16:56.847 DEBUG:boto:Token: None 2024-03-04T16:16:56.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/?versions 2024-03-04T16:16:56.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA= 2024-03-04T16:16:56.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA=', 'Content-Length': '0'} 2024-03-04T16:16:56.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f387434f9c8eef69-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.851 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.851 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:56.851 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:56.851 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:56.851 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:56.852 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:56.852 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:56.852 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:56.852 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:56.852 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:56.852 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.852 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.852 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.852 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.852 DEBUG:boto:Method: GET 2024-03-04T16:16:56.852 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.853 DEBUG:boto:Data: 2024-03-04T16:16:56.853 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.853 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.853 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.853 DEBUG:boto:Params: {} 2024-03-04T16:16:56.853 DEBUG:boto:Token: None 2024-03-04T16:16:56.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/?versions 2024-03-04T16:16:56.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA= 2024-03-04T16:16:56.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA=', 'Content-Length': '0'} 2024-03-04T16:16:56.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84b63c679dcc8d1-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.855 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.856 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.856 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.856 DEBUG:boto:path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.856 DEBUG:boto:auth_path=/sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.856 DEBUG:boto:Method: GET 2024-03-04T16:16:56.856 DEBUG:boto:Path: /sbnvju-11/?versions&max-keys=999 2024-03-04T16:16:56.857 DEBUG:boto:Data: 2024-03-04T16:16:56.857 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.857 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.857 DEBUG:boto:Params: {} 2024-03-04T16:16:56.857 DEBUG:boto:Token: None 2024-03-04T16:16:56.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/?versions 2024-03-04T16:16:56.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA= 2024-03-04T16:16:56.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1pkotypC7MpEfmUP37JY1pkxSEA=', 'Content-Length': '0'} 2024-03-04T16:16:56.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c12ce191ce8b972-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.860 DEBUG:boto:b'sbnvju-11999false&nulltrue2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:56.861 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.861 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.861 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.861 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.861 DEBUG:boto:Method: GET 2024-03-04T16:16:56.861 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.861 DEBUG:boto:Data: 2024-03-04T16:16:56.861 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.861 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.861 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.862 DEBUG:boto:Params: {} 2024-03-04T16:16:56.862 DEBUG:boto:Token: None 2024-03-04T16:16:56.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU= 2024-03-04T16:16:56.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU=', 'Content-Length': '0'} 2024-03-04T16:16:56.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfbf5ecafad6ad43-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.864 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.864 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.864 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.864 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.865 DEBUG:boto:Method: GET 2024-03-04T16:16:56.865 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.865 DEBUG:boto:Data: 2024-03-04T16:16:56.865 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.865 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.865 DEBUG:boto:Params: {} 2024-03-04T16:16:56.865 DEBUG:boto:Token: None 2024-03-04T16:16:56.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.866 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU= 2024-03-04T16:16:56.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU=', 'Content-Length': '0'} 2024-03-04T16:16:56.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000233b75251c227321-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.868 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.868 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.868 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.868 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.868 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.868 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.868 DEBUG:boto:Data: 2024-03-04T16:16:56.868 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.868 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.869 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.869 DEBUG:boto:Params: {} 2024-03-04T16:16:56.869 DEBUG:boto:Token: None 2024-03-04T16:16:56.869 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.869 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UOgsKx0wNT9z5zVoTOIgw14Kl8k= 2024-03-04T16:16:56.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UOgsKx0wNT9z5zVoTOIgw14Kl8k=', 'Content-Length': '0'} 2024-03-04T16:16:56.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037458db811a0db2d-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.871 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.871 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.871 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.871 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.871 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.871 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.871 DEBUG:boto:Data: 2024-03-04T16:16:56.871 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.871 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.872 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.872 DEBUG:boto:Params: {} 2024-03-04T16:16:56.872 DEBUG:boto:Token: None 2024-03-04T16:16:56.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.872 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UOgsKx0wNT9z5zVoTOIgw14Kl8k= 2024-03-04T16:16:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UOgsKx0wNT9z5zVoTOIgw14Kl8k=', 'Content-Length': '0'} 2024-03-04T16:16:56.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041ebabd7270ffb5c-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:56.874 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.874 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.875 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.875 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.875 DEBUG:boto:Method: GET 2024-03-04T16:16:56.875 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.875 DEBUG:boto:Data: 2024-03-04T16:16:56.875 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.875 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.875 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.875 DEBUG:boto:Params: {} 2024-03-04T16:16:56.875 DEBUG:boto:Token: None 2024-03-04T16:16:56.875 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.876 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU= 2024-03-04T16:16:56.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU=', 'Content-Length': '0'} 2024-03-04T16:16:56.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035dd2948be0f04a6-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.920 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.920 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.920 DEBUG:boto:path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.920 DEBUG:boto:auth_path=/sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.921 DEBUG:boto:Method: GET 2024-03-04T16:16:56.921 DEBUG:boto:Path: /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.921 DEBUG:boto:Data: 2024-03-04T16:16:56.921 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.921 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.921 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.921 DEBUG:boto:Params: {} 2024-03-04T16:16:56.921 DEBUG:boto:Token: None 2024-03-04T16:16:56.921 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26?versionId=null 2024-03-04T16:16:56.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU= 2024-03-04T16:16:56.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FVrDpnKeYjfGbikdI4BXMf8RnsU=', 'Content-Length': '0'} 2024-03-04T16:16:56.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b35d096c74da956-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.924 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.924 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.924 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:56.924 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:56.924 DEBUG:boto:Method: GET 2024-03-04T16:16:56.924 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:56.924 DEBUG:boto:Data: 2024-03-04T16:16:56.925 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.925 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.925 DEBUG:boto:Params: {} 2024-03-04T16:16:56.925 DEBUG:boto:Token: None 2024-03-04T16:16:56.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.925 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000549f91371d707c4b-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.968 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.968 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:56.968 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:56.969 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:56.969 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:56.969 DEBUG:boto:Method: GET 2024-03-04T16:16:56.969 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:56.969 DEBUG:boto:Data: 2024-03-04T16:16:56.969 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.969 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.969 DEBUG:boto:Params: {} 2024-03-04T16:16:56.969 DEBUG:boto:Token: None 2024-03-04T16:16:56.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/ 2024-03-04T16:16:56.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI= 2024-03-04T16:16:56.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L9bN7at3aCZVoGtqgb9kxR1q0gI=', 'Content-Length': '0'} 2024-03-04T16:16:56.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6adc3611971c9b-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.972 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:56.973 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.973 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.973 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.973 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:56.973 DEBUG:boto:Data: 2024-03-04T16:16:56.973 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.973 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.973 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.973 DEBUG:boto:Params: {} 2024-03-04T16:16:56.974 DEBUG:boto:Token: None 2024-03-04T16:16:56.974 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26 2024-03-04T16:16:56.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ub5ap3coz8s6fROX4g6+Ue2rOGg= 2024-03-04T16:16:56.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ub5ap3coz8s6fROX4g6+Ue2rOGg=', 'Content-Length': '0'} 2024-03-04T16:16:56.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd41447c5aecf4ed-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.976 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.976 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.976 DEBUG:boto:Method: HEAD 2024-03-04T16:16:56.976 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:56.976 DEBUG:boto:Data: 2024-03-04T16:16:56.976 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.976 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:56.976 DEBUG:boto:Port: 8000 2024-03-04T16:16:56.976 DEBUG:boto:Params: {} 2024-03-04T16:16:56.976 DEBUG:boto:Token: None 2024-03-04T16:16:56.977 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26 2024-03-04T16:16:56.977 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ub5ap3coz8s6fROX4g6+Ue2rOGg= 2024-03-04T16:16:56.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ub5ap3coz8s6fROX4g6+Ue2rOGg=', 'Content-Length': '0'} 2024-03-04T16:16:56.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0d167898486b0ed-0065e5f3f8-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:56.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:56.979 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:56.979 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:56.979 DEBUG:boto:Method: GET 2024-03-04T16:16:56.979 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:56.979 DEBUG:boto:Data: 2024-03-04T16:16:56.979 DEBUG:boto:Headers: {} 2024-03-04T16:16:56.979 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:56.980 DEBUG:boto:Port: 8001 2024-03-04T16:16:56.980 DEBUG:boto:Params: {} 2024-03-04T16:16:56.980 DEBUG:boto:Token: None 2024-03-04T16:16:56.980 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:56 GMT /sbnvju-11/%26 2024-03-04T16:16:56.980 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cSAysyxJKRkugOYiA1cYAuu7t1w= 2024-03-04T16:16:56.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cSAysyxJKRkugOYiA1cYAuu7t1w=', 'Content-Length': '0'} 2024-03-04T16:16:56.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ecddeb8c07b6db9-0065e5f3f8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.024 DEBUG:boto:path=/sbnvju-11/%26 2024-03-04T16:16:57.024 DEBUG:boto:auth_path=/sbnvju-11/%26 2024-03-04T16:16:57.024 DEBUG:boto:Method: GET 2024-03-04T16:16:57.024 DEBUG:boto:Path: /sbnvju-11/%26 2024-03-04T16:16:57.024 DEBUG:boto:Data: 2024-03-04T16:16:57.025 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.025 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.025 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.025 DEBUG:boto:Params: {} 2024-03-04T16:16:57.025 DEBUG:boto:Token: None 2024-03-04T16:16:57.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/%26 2024-03-04T16:16:57.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BTArvKQ9RDPANRXfIb3yw8D65Zg= 2024-03-04T16:16:57.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BTArvKQ9RDPANRXfIb3yw8D65Zg=', 'Content-Length': '0'} 2024-03-04T16:16:57.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ba9130c83a15252-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.027 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.027 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.027 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:57.027 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:57.028 DEBUG:boto:Method: GET 2024-03-04T16:16:57.028 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:57.028 DEBUG:boto:Data: 2024-03-04T16:16:57.028 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.028 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.028 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.028 DEBUG:boto:Params: {} 2024-03-04T16:16:57.028 DEBUG:boto:Token: None 2024-03-04T16:16:57.028 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.028 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf4bca45b487d77-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.031 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.031 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.031 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.031 DEBUG:boto:path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:57.032 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%26 2024-03-04T16:16:57.032 DEBUG:boto:Method: GET 2024-03-04T16:16:57.032 DEBUG:boto:Path: /sbnvju-11/?prefix=%26 2024-03-04T16:16:57.032 DEBUG:boto:Data: 2024-03-04T16:16:57.032 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.032 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.032 DEBUG:boto:Params: {} 2024-03-04T16:16:57.032 DEBUG:boto:Token: None 2024-03-04T16:16:57.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.033 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c023c347db2d2c-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.035 DEBUG:boto:b'sbnvju-11&1000false&2024-03-04T16:13:01.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:57.036 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.036 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.036 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.036 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.036 DEBUG:boto:Method: GET 2024-03-04T16:16:57.036 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.036 DEBUG:boto:Data: 2024-03-04T16:16:57.036 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.036 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.037 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.037 DEBUG:boto:Params: {} 2024-03-04T16:16:57.037 DEBUG:boto:Token: None 2024-03-04T16:16:57.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.037 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk= 2024-03-04T16:16:57.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk=', 'Content-Length': '0'} 2024-03-04T16:16:57.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd82a756c36905ca-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.039 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.039 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.039 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.039 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.040 DEBUG:boto:Method: GET 2024-03-04T16:16:57.040 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.040 DEBUG:boto:Data: 2024-03-04T16:16:57.040 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.040 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.040 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.040 DEBUG:boto:Params: {} 2024-03-04T16:16:57.040 DEBUG:boto:Token: None 2024-03-04T16:16:57.040 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.040 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk= 2024-03-04T16:16:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk=', 'Content-Length': '0'} 2024-03-04T16:16:57.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000515a2a4fa7b84385-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.042 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.043 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.043 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.043 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.043 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.043 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.043 DEBUG:boto:Data: 2024-03-04T16:16:57.043 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.043 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.043 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.043 DEBUG:boto:Params: {} 2024-03-04T16:16:57.044 DEBUG:boto:Token: None 2024-03-04T16:16:57.044 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.044 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cGQj1P3Aay5yBrSfnifOHu/nc2k= 2024-03-04T16:16:57.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cGQj1P3Aay5yBrSfnifOHu/nc2k=', 'Content-Length': '0'} 2024-03-04T16:16:57.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4eececfe0deacd8-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.046 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.046 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.046 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.046 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.046 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.046 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.046 DEBUG:boto:Data: 2024-03-04T16:16:57.046 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.046 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.047 DEBUG:boto:Params: {} 2024-03-04T16:16:57.047 DEBUG:boto:Token: None 2024-03-04T16:16:57.047 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.047 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cGQj1P3Aay5yBrSfnifOHu/nc2k= 2024-03-04T16:16:57.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cGQj1P3Aay5yBrSfnifOHu/nc2k=', 'Content-Length': '0'} 2024-03-04T16:16:57.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000458ee4783a38c0a1-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:57.049 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.049 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.049 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.049 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.049 DEBUG:boto:Method: GET 2024-03-04T16:16:57.050 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.050 DEBUG:boto:Data: 2024-03-04T16:16:57.050 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.050 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.050 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.050 DEBUG:boto:Params: {} 2024-03-04T16:16:57.050 DEBUG:boto:Token: None 2024-03-04T16:16:57.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk= 2024-03-04T16:16:57.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk=', 'Content-Length': '0'} 2024-03-04T16:16:57.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002717ebf48aa8fd92-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.092 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.092 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.092 DEBUG:boto:path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.092 DEBUG:boto:auth_path=/sbnvju-11/.?versionId=null 2024-03-04T16:16:57.092 DEBUG:boto:Method: GET 2024-03-04T16:16:57.093 DEBUG:boto:Path: /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.093 DEBUG:boto:Data: 2024-03-04T16:16:57.093 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.093 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.093 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.093 DEBUG:boto:Params: {} 2024-03-04T16:16:57.093 DEBUG:boto:Token: None 2024-03-04T16:16:57.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.?versionId=null 2024-03-04T16:16:57.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk= 2024-03-04T16:16:57.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3JVMGaL7wjlc2wM1dZWUz9D4Tk=', 'Content-Length': '0'} 2024-03-04T16:16:57.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e77363bcf3684b2d-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.095 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.096 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.096 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.096 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.096 DEBUG:boto:Method: GET 2024-03-04T16:16:57.096 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:57.096 DEBUG:boto:Data: 2024-03-04T16:16:57.096 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.096 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.096 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.096 DEBUG:boto:Params: {} 2024-03-04T16:16:57.097 DEBUG:boto:Token: None 2024-03-04T16:16:57.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.097 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d93e50b3af78500-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.099 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.100 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.100 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.100 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.100 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.100 DEBUG:boto:Method: GET 2024-03-04T16:16:57.100 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:57.100 DEBUG:boto:Data: 2024-03-04T16:16:57.100 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.100 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.101 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.101 DEBUG:boto:Params: {} 2024-03-04T16:16:57.101 DEBUG:boto:Token: None 2024-03-04T16:16:57.101 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.101 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84251b933015ce8-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.104 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.104 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.104 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.104 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.104 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:57.105 DEBUG:boto:Data: 2024-03-04T16:16:57.105 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.105 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.105 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.105 DEBUG:boto:Params: {} 2024-03-04T16:16:57.105 DEBUG:boto:Token: None 2024-03-04T16:16:57.105 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/. 2024-03-04T16:16:57.105 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7+QnkIjcd94up6xI/SJeRNS/3bc= 2024-03-04T16:16:57.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7+QnkIjcd94up6xI/SJeRNS/3bc=', 'Content-Length': '0'} 2024-03-04T16:16:57.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021a388f7e85ce3d7-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.107 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.107 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.107 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.107 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:57.107 DEBUG:boto:Data: 2024-03-04T16:16:57.107 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.108 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.108 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.108 DEBUG:boto:Params: {} 2024-03-04T16:16:57.108 DEBUG:boto:Token: None 2024-03-04T16:16:57.108 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/. 2024-03-04T16:16:57.108 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7+QnkIjcd94up6xI/SJeRNS/3bc= 2024-03-04T16:16:57.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7+QnkIjcd94up6xI/SJeRNS/3bc=', 'Content-Length': '0'} 2024-03-04T16:16:57.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062b1681f4f91a36f-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:57.110 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.111 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.111 DEBUG:boto:Method: GET 2024-03-04T16:16:57.111 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:57.111 DEBUG:boto:Data: 2024-03-04T16:16:57.111 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.111 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.111 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.111 DEBUG:boto:Params: {} 2024-03-04T16:16:57.111 DEBUG:boto:Token: None 2024-03-04T16:16:57.111 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/. 2024-03-04T16:16:57.111 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3fYqE2RGA8gHHpwc9ZL2CIMTxx8= 2024-03-04T16:16:57.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3fYqE2RGA8gHHpwc9ZL2CIMTxx8=', 'Content-Length': '0'} 2024-03-04T16:16:57.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b480a23f0037e98c-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.160 DEBUG:boto:path=/sbnvju-11/. 2024-03-04T16:16:57.160 DEBUG:boto:auth_path=/sbnvju-11/. 2024-03-04T16:16:57.160 DEBUG:boto:Method: GET 2024-03-04T16:16:57.160 DEBUG:boto:Path: /sbnvju-11/. 2024-03-04T16:16:57.160 DEBUG:boto:Data: 2024-03-04T16:16:57.160 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.161 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.161 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.161 DEBUG:boto:Params: {} 2024-03-04T16:16:57.161 DEBUG:boto:Token: None 2024-03-04T16:16:57.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/. 2024-03-04T16:16:57.161 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3fYqE2RGA8gHHpwc9ZL2CIMTxx8= 2024-03-04T16:16:57.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3fYqE2RGA8gHHpwc9ZL2CIMTxx8=', 'Content-Length': '0'} 2024-03-04T16:16:57.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009092316d998fb1fc-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.163 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.163 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.163 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.164 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.164 DEBUG:boto:Method: GET 2024-03-04T16:16:57.164 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:57.164 DEBUG:boto:Data: 2024-03-04T16:16:57.164 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.164 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.164 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.164 DEBUG:boto:Params: {} 2024-03-04T16:16:57.164 DEBUG:boto:Token: None 2024-03-04T16:16:57.164 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4241043e6c2fa28-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.167 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.168 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.168 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.168 DEBUG:boto:path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.168 DEBUG:boto:auth_path=/sbnvju-11/?prefix=. 2024-03-04T16:16:57.168 DEBUG:boto:Method: GET 2024-03-04T16:16:57.168 DEBUG:boto:Path: /sbnvju-11/?prefix=. 2024-03-04T16:16:57.168 DEBUG:boto:Data: 2024-03-04T16:16:57.168 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.168 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.168 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.168 DEBUG:boto:Params: {} 2024-03-04T16:16:57.169 DEBUG:boto:Token: None 2024-03-04T16:16:57.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077aeedcc9c42259e-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.172 DEBUG:boto:b'sbnvju-11.1000false.2024-03-04T16:13:01.463Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:57.172 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.172 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.172 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.172 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.172 DEBUG:boto:Method: GET 2024-03-04T16:16:57.173 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.173 DEBUG:boto:Data: 2024-03-04T16:16:57.173 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.173 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.173 DEBUG:boto:Params: {} 2024-03-04T16:16:57.173 DEBUG:boto:Token: None 2024-03-04T16:16:57.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4= 2024-03-04T16:16:57.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4=', 'Content-Length': '0'} 2024-03-04T16:16:57.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000145be21ce74489ea-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.175 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.175 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.175 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.175 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.175 DEBUG:boto:Method: GET 2024-03-04T16:16:57.175 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.176 DEBUG:boto:Data: 2024-03-04T16:16:57.176 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.176 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.176 DEBUG:boto:Params: {} 2024-03-04T16:16:57.176 DEBUG:boto:Token: None 2024-03-04T16:16:57.176 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.176 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4= 2024-03-04T16:16:57.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4=', 'Content-Length': '0'} 2024-03-04T16:16:57.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026d125993eb2d456-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.179 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.179 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.179 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.179 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.179 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.179 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.179 DEBUG:boto:Data: 2024-03-04T16:16:57.179 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.179 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.179 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.179 DEBUG:boto:Params: {} 2024-03-04T16:16:57.180 DEBUG:boto:Token: None 2024-03-04T16:16:57.180 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.180 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KUMAZkkOju7Arn47Y+bsCJJMrt8= 2024-03-04T16:16:57.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KUMAZkkOju7Arn47Y+bsCJJMrt8=', 'Content-Length': '0'} 2024-03-04T16:16:57.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bef795b9e559416-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.182 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.182 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.182 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.182 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.182 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.182 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.182 DEBUG:boto:Data: 2024-03-04T16:16:57.182 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.182 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.183 DEBUG:boto:Params: {} 2024-03-04T16:16:57.183 DEBUG:boto:Token: None 2024-03-04T16:16:57.183 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.183 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KUMAZkkOju7Arn47Y+bsCJJMrt8= 2024-03-04T16:16:57.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KUMAZkkOju7Arn47Y+bsCJJMrt8=', 'Content-Length': '0'} 2024-03-04T16:16:57.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d656ab124e9b5731-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:57.185 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.185 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.185 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.186 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.186 DEBUG:boto:Method: GET 2024-03-04T16:16:57.186 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.186 DEBUG:boto:Data: 2024-03-04T16:16:57.186 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.186 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.186 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.186 DEBUG:boto:Params: {} 2024-03-04T16:16:57.186 DEBUG:boto:Token: None 2024-03-04T16:16:57.186 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.187 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4= 2024-03-04T16:16:57.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4=', 'Content-Length': '0'} 2024-03-04T16:16:57.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055acbacaa784fc82-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.232 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.232 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.232 DEBUG:boto:path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.232 DEBUG:boto:auth_path=/sbnvju-11/..?versionId=null 2024-03-04T16:16:57.233 DEBUG:boto:Method: GET 2024-03-04T16:16:57.233 DEBUG:boto:Path: /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.233 DEBUG:boto:Data: 2024-03-04T16:16:57.233 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.233 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.233 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.233 DEBUG:boto:Params: {} 2024-03-04T16:16:57.233 DEBUG:boto:Token: None 2024-03-04T16:16:57.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/..?versionId=null 2024-03-04T16:16:57.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4= 2024-03-04T16:16:57.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y0TQOQH1QVNITi/XGtYuG5wCNu4=', 'Content-Length': '0'} 2024-03-04T16:16:57.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014f56ca4a589ec32-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.236 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.236 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.236 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.236 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.236 DEBUG:boto:Method: GET 2024-03-04T16:16:57.236 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:57.236 DEBUG:boto:Data: 2024-03-04T16:16:57.236 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.236 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.236 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.237 DEBUG:boto:Params: {} 2024-03-04T16:16:57.237 DEBUG:boto:Token: None 2024-03-04T16:16:57.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f587de8500282f6e-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.239 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.240 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.240 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.240 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.240 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.240 DEBUG:boto:Method: GET 2024-03-04T16:16:57.240 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:57.240 DEBUG:boto:Data: 2024-03-04T16:16:57.240 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.240 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.240 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.240 DEBUG:boto:Params: {} 2024-03-04T16:16:57.240 DEBUG:boto:Token: None 2024-03-04T16:16:57.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ad34438fd3024a-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.244 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.244 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.244 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.244 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.244 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:57.244 DEBUG:boto:Data: 2024-03-04T16:16:57.244 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.244 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.244 DEBUG:boto:Params: {} 2024-03-04T16:16:57.245 DEBUG:boto:Token: None 2024-03-04T16:16:57.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.. 2024-03-04T16:16:57.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mVXyKXRheKg+WrPPFkHVkmt/770= 2024-03-04T16:16:57.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mVXyKXRheKg+WrPPFkHVkmt/770=', 'Content-Length': '0'} 2024-03-04T16:16:57.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000647174a8177f5fd5-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.247 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.247 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.247 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.247 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:57.247 DEBUG:boto:Data: 2024-03-04T16:16:57.247 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.247 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.247 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.247 DEBUG:boto:Params: {} 2024-03-04T16:16:57.247 DEBUG:boto:Token: None 2024-03-04T16:16:57.247 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.. 2024-03-04T16:16:57.248 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mVXyKXRheKg+WrPPFkHVkmt/770= 2024-03-04T16:16:57.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mVXyKXRheKg+WrPPFkHVkmt/770=', 'Content-Length': '0'} 2024-03-04T16:16:57.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004141632ad2a6b350-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:57.250 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.250 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.250 DEBUG:boto:Method: GET 2024-03-04T16:16:57.250 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:57.251 DEBUG:boto:Data: 2024-03-04T16:16:57.251 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.251 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.251 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.251 DEBUG:boto:Params: {} 2024-03-04T16:16:57.251 DEBUG:boto:Token: None 2024-03-04T16:16:57.251 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.. 2024-03-04T16:16:57.251 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yBZVTyqKDr0abBa+0kY6IHPyekU= 2024-03-04T16:16:57.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yBZVTyqKDr0abBa+0kY6IHPyekU=', 'Content-Length': '0'} 2024-03-04T16:16:57.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a7746298f61d0e1-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.296 DEBUG:boto:path=/sbnvju-11/.. 2024-03-04T16:16:57.296 DEBUG:boto:auth_path=/sbnvju-11/.. 2024-03-04T16:16:57.296 DEBUG:boto:Method: GET 2024-03-04T16:16:57.296 DEBUG:boto:Path: /sbnvju-11/.. 2024-03-04T16:16:57.296 DEBUG:boto:Data: 2024-03-04T16:16:57.296 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.297 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.297 DEBUG:boto:Params: {} 2024-03-04T16:16:57.297 DEBUG:boto:Token: None 2024-03-04T16:16:57.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.. 2024-03-04T16:16:57.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yBZVTyqKDr0abBa+0kY6IHPyekU= 2024-03-04T16:16:57.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yBZVTyqKDr0abBa+0kY6IHPyekU=', 'Content-Length': '0'} 2024-03-04T16:16:57.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006a2a2caa450f67d-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.299 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.299 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.300 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.300 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.300 DEBUG:boto:Method: GET 2024-03-04T16:16:57.300 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:57.300 DEBUG:boto:Data: 2024-03-04T16:16:57.300 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.300 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.300 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.300 DEBUG:boto:Params: {} 2024-03-04T16:16:57.300 DEBUG:boto:Token: None 2024-03-04T16:16:57.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.301 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e289e5420c4fcd-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.344 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.344 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.344 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.345 DEBUG:boto:path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.345 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.. 2024-03-04T16:16:57.345 DEBUG:boto:Method: GET 2024-03-04T16:16:57.345 DEBUG:boto:Path: /sbnvju-11/?prefix=.. 2024-03-04T16:16:57.345 DEBUG:boto:Data: 2024-03-04T16:16:57.345 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.345 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.345 DEBUG:boto:Params: {} 2024-03-04T16:16:57.345 DEBUG:boto:Token: None 2024-03-04T16:16:57.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.346 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f940f699cafbe852-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.348 DEBUG:boto:b'sbnvju-11..1000false..2024-03-04T16:13:01.477Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:57.349 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.349 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.349 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.349 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.349 DEBUG:boto:Method: GET 2024-03-04T16:16:57.349 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.349 DEBUG:boto:Data: 2024-03-04T16:16:57.350 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.350 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.350 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.350 DEBUG:boto:Params: {} 2024-03-04T16:16:57.350 DEBUG:boto:Token: None 2024-03-04T16:16:57.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY= 2024-03-04T16:16:57.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY=', 'Content-Length': '0'} 2024-03-04T16:16:57.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf8b16cf8ae3d9ca-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.352 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.352 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.352 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.352 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.353 DEBUG:boto:Method: GET 2024-03-04T16:16:57.353 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.353 DEBUG:boto:Data: 2024-03-04T16:16:57.353 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.353 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.353 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.353 DEBUG:boto:Params: {} 2024-03-04T16:16:57.353 DEBUG:boto:Token: None 2024-03-04T16:16:57.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY= 2024-03-04T16:16:57.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY=', 'Content-Length': '0'} 2024-03-04T16:16:57.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000913f32fad35d8e15-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.396 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.396 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.396 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.396 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.396 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.397 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.397 DEBUG:boto:Data: 2024-03-04T16:16:57.397 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.397 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.397 DEBUG:boto:Params: {} 2024-03-04T16:16:57.397 DEBUG:boto:Token: None 2024-03-04T16:16:57.397 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J2UEO7UxUEwHRhHsr2myWDVwERs= 2024-03-04T16:16:57.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J2UEO7UxUEwHRhHsr2myWDVwERs=', 'Content-Length': '0'} 2024-03-04T16:16:57.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000176076e51ebb2f1-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.399 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.400 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.400 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.400 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.400 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.400 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.400 DEBUG:boto:Data: 2024-03-04T16:16:57.400 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.400 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.400 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.400 DEBUG:boto:Params: {} 2024-03-04T16:16:57.400 DEBUG:boto:Token: None 2024-03-04T16:16:57.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J2UEO7UxUEwHRhHsr2myWDVwERs= 2024-03-04T16:16:57.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J2UEO7UxUEwHRhHsr2myWDVwERs=', 'Content-Length': '0'} 2024-03-04T16:16:57.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067b41e6ca5a4037a-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:57.403 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.403 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.403 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.403 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.403 DEBUG:boto:Method: GET 2024-03-04T16:16:57.403 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.403 DEBUG:boto:Data: 2024-03-04T16:16:57.403 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.403 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.404 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.404 DEBUG:boto:Params: {} 2024-03-04T16:16:57.404 DEBUG:boto:Token: None 2024-03-04T16:16:57.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.404 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY= 2024-03-04T16:16:57.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY=', 'Content-Length': '0'} 2024-03-04T16:16:57.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070e41df55d470581-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.448 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.448 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.448 DEBUG:boto:path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.448 DEBUG:boto:auth_path=/sbnvju-11/...?versionId=null 2024-03-04T16:16:57.449 DEBUG:boto:Method: GET 2024-03-04T16:16:57.449 DEBUG:boto:Path: /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.449 DEBUG:boto:Data: 2024-03-04T16:16:57.449 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.449 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.449 DEBUG:boto:Params: {} 2024-03-04T16:16:57.449 DEBUG:boto:Token: None 2024-03-04T16:16:57.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/...?versionId=null 2024-03-04T16:16:57.449 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY= 2024-03-04T16:16:57.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Brcyt2qmxqHRCpoVqe/mjVJowCY=', 'Content-Length': '0'} 2024-03-04T16:16:57.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e893ac60cf54f5-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.452 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.452 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.452 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.452 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.452 DEBUG:boto:Method: GET 2024-03-04T16:16:57.452 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:57.452 DEBUG:boto:Data: 2024-03-04T16:16:57.452 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.452 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.452 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.452 DEBUG:boto:Params: {} 2024-03-04T16:16:57.453 DEBUG:boto:Token: None 2024-03-04T16:16:57.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4f68bf858fff3f-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.455 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.456 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.456 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.456 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.456 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.456 DEBUG:boto:Method: GET 2024-03-04T16:16:57.456 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:57.456 DEBUG:boto:Data: 2024-03-04T16:16:57.456 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.456 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.456 DEBUG:boto:Params: {} 2024-03-04T16:16:57.457 DEBUG:boto:Token: None 2024-03-04T16:16:57.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5221021aa1d8b59-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.459 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.460 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.460 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.460 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.460 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:57.460 DEBUG:boto:Data: 2024-03-04T16:16:57.460 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.460 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.460 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.460 DEBUG:boto:Params: {} 2024-03-04T16:16:57.460 DEBUG:boto:Token: None 2024-03-04T16:16:57.461 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/... 2024-03-04T16:16:57.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TeALJ5JSew1EaIkRX9c98Whyp9g= 2024-03-04T16:16:57.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TeALJ5JSew1EaIkRX9c98Whyp9g=', 'Content-Length': '0'} 2024-03-04T16:16:57.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca07e2556f1c8cef-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.463 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.463 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.463 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.463 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:57.463 DEBUG:boto:Data: 2024-03-04T16:16:57.463 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.463 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.463 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.464 DEBUG:boto:Params: {} 2024-03-04T16:16:57.464 DEBUG:boto:Token: None 2024-03-04T16:16:57.464 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/... 2024-03-04T16:16:57.464 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TeALJ5JSew1EaIkRX9c98Whyp9g= 2024-03-04T16:16:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TeALJ5JSew1EaIkRX9c98Whyp9g=', 'Content-Length': '0'} 2024-03-04T16:16:57.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba0fadbcce72bbac-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:57.466 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.466 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.466 DEBUG:boto:Method: GET 2024-03-04T16:16:57.466 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:57.467 DEBUG:boto:Data: 2024-03-04T16:16:57.467 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.467 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.467 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.467 DEBUG:boto:Params: {} 2024-03-04T16:16:57.467 DEBUG:boto:Token: None 2024-03-04T16:16:57.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/... 2024-03-04T16:16:57.467 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rwiRzWdX2F+EyCKoFIXPjnK5aL0= 2024-03-04T16:16:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rwiRzWdX2F+EyCKoFIXPjnK5aL0=', 'Content-Length': '0'} 2024-03-04T16:16:57.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e19265db1e743bd1-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.512 DEBUG:boto:path=/sbnvju-11/... 2024-03-04T16:16:57.512 DEBUG:boto:auth_path=/sbnvju-11/... 2024-03-04T16:16:57.512 DEBUG:boto:Method: GET 2024-03-04T16:16:57.512 DEBUG:boto:Path: /sbnvju-11/... 2024-03-04T16:16:57.512 DEBUG:boto:Data: 2024-03-04T16:16:57.512 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.513 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.513 DEBUG:boto:Params: {} 2024-03-04T16:16:57.513 DEBUG:boto:Token: None 2024-03-04T16:16:57.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/... 2024-03-04T16:16:57.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rwiRzWdX2F+EyCKoFIXPjnK5aL0= 2024-03-04T16:16:57.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rwiRzWdX2F+EyCKoFIXPjnK5aL0=', 'Content-Length': '0'} 2024-03-04T16:16:57.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb1e35ec0c572def-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.515 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.515 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.515 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.515 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.515 DEBUG:boto:Method: GET 2024-03-04T16:16:57.516 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:57.516 DEBUG:boto:Data: 2024-03-04T16:16:57.516 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.516 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.516 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.516 DEBUG:boto:Params: {} 2024-03-04T16:16:57.516 DEBUG:boto:Token: None 2024-03-04T16:16:57.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.516 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6324e63c8aa4aec-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.560 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.560 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.560 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.560 DEBUG:boto:path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.561 DEBUG:boto:auth_path=/sbnvju-11/?prefix=... 2024-03-04T16:16:57.561 DEBUG:boto:Method: GET 2024-03-04T16:16:57.561 DEBUG:boto:Path: /sbnvju-11/?prefix=... 2024-03-04T16:16:57.561 DEBUG:boto:Data: 2024-03-04T16:16:57.561 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.561 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.561 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.561 DEBUG:boto:Params: {} 2024-03-04T16:16:57.561 DEBUG:boto:Token: None 2024-03-04T16:16:57.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.561 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8eedf479151e23b-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.564 DEBUG:boto:b'sbnvju-11...1000false...2024-03-04T16:13:01.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:57.564 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.565 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.565 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.565 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.565 DEBUG:boto:Method: GET 2024-03-04T16:16:57.565 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.565 DEBUG:boto:Data: 2024-03-04T16:16:57.565 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.565 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.565 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.565 DEBUG:boto:Params: {} 2024-03-04T16:16:57.565 DEBUG:boto:Token: None 2024-03-04T16:16:57.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY= 2024-03-04T16:16:57.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY=', 'Content-Length': '0'} 2024-03-04T16:16:57.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000546991ba6c17ec97-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.608 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.608 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.608 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.608 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.608 DEBUG:boto:Method: GET 2024-03-04T16:16:57.608 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.609 DEBUG:boto:Data: 2024-03-04T16:16:57.609 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.609 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.609 DEBUG:boto:Params: {} 2024-03-04T16:16:57.609 DEBUG:boto:Token: None 2024-03-04T16:16:57.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.609 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY= 2024-03-04T16:16:57.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY=', 'Content-Length': '0'} 2024-03-04T16:16:57.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f5943f89d972751-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.612 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.612 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.612 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.612 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.612 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.612 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.612 DEBUG:boto:Data: 2024-03-04T16:16:57.612 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.612 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.612 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.612 DEBUG:boto:Params: {} 2024-03-04T16:16:57.613 DEBUG:boto:Token: None 2024-03-04T16:16:57.613 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.613 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NxN5iv7GxulJE+roBYN/ZgtnQUs= 2024-03-04T16:16:57.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NxN5iv7GxulJE+roBYN/ZgtnQUs=', 'Content-Length': '0'} 2024-03-04T16:16:57.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb06d61cef8e7e98-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.615 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.615 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.615 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.615 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.616 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.616 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.616 DEBUG:boto:Data: 2024-03-04T16:16:57.616 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.616 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.616 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.616 DEBUG:boto:Params: {} 2024-03-04T16:16:57.616 DEBUG:boto:Token: None 2024-03-04T16:16:57.616 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.617 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NxN5iv7GxulJE+roBYN/ZgtnQUs= 2024-03-04T16:16:57.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NxN5iv7GxulJE+roBYN/ZgtnQUs=', 'Content-Length': '0'} 2024-03-04T16:16:57.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ccdeeb614af77fc-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:57.619 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.619 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.619 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.619 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.619 DEBUG:boto:Method: GET 2024-03-04T16:16:57.620 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.620 DEBUG:boto:Data: 2024-03-04T16:16:57.620 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.620 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.620 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.620 DEBUG:boto:Params: {} 2024-03-04T16:16:57.620 DEBUG:boto:Token: None 2024-03-04T16:16:57.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.620 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY= 2024-03-04T16:16:57.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY=', 'Content-Length': '0'} 2024-03-04T16:16:57.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba022fd7631fe6c7-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.664 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.664 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.664 DEBUG:boto:path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.664 DEBUG:boto:auth_path=/sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.664 DEBUG:boto:Method: GET 2024-03-04T16:16:57.664 DEBUG:boto:Path: /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.665 DEBUG:boto:Data: 2024-03-04T16:16:57.665 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.665 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.665 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.665 DEBUG:boto:Params: {} 2024-03-04T16:16:57.665 DEBUG:boto:Token: None 2024-03-04T16:16:57.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o?versionId=null 2024-03-04T16:16:57.665 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY= 2024-03-04T16:16:57.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:33JrU8VozKYk+zcthRxoswr2EdY=', 'Content-Length': '0'} 2024-03-04T16:16:57.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b63f31ee04d400ec-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.668 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.668 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.668 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.668 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.669 DEBUG:boto:Method: GET 2024-03-04T16:16:57.669 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:57.669 DEBUG:boto:Data: 2024-03-04T16:16:57.669 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.669 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.669 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.669 DEBUG:boto:Params: {} 2024-03-04T16:16:57.669 DEBUG:boto:Token: None 2024-03-04T16:16:57.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe95cae2be22a5e4-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.712 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.712 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.713 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.713 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.713 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.713 DEBUG:boto:Method: GET 2024-03-04T16:16:57.713 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:57.713 DEBUG:boto:Data: 2024-03-04T16:16:57.713 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.713 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.713 DEBUG:boto:Params: {} 2024-03-04T16:16:57.713 DEBUG:boto:Token: None 2024-03-04T16:16:57.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178db00b500b1fa2-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.716 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.717 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.717 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.717 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.717 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:57.717 DEBUG:boto:Data: 2024-03-04T16:16:57.717 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.717 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.717 DEBUG:boto:Params: {} 2024-03-04T16:16:57.717 DEBUG:boto:Token: None 2024-03-04T16:16:57.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o 2024-03-04T16:16:57.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yG3fW7dktcvGmEcGV7ld8K+tVXQ= 2024-03-04T16:16:57.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yG3fW7dktcvGmEcGV7ld8K+tVXQ=', 'Content-Length': '0'} 2024-03-04T16:16:57.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003834175567003c29-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.720 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.720 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.720 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.720 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:57.720 DEBUG:boto:Data: 2024-03-04T16:16:57.720 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.720 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.720 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.720 DEBUG:boto:Params: {} 2024-03-04T16:16:57.720 DEBUG:boto:Token: None 2024-03-04T16:16:57.721 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o 2024-03-04T16:16:57.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yG3fW7dktcvGmEcGV7ld8K+tVXQ= 2024-03-04T16:16:57.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yG3fW7dktcvGmEcGV7ld8K+tVXQ=', 'Content-Length': '0'} 2024-03-04T16:16:57.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003edd799b01403b95-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:57.723 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.723 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.723 DEBUG:boto:Method: GET 2024-03-04T16:16:57.723 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:57.723 DEBUG:boto:Data: 2024-03-04T16:16:57.724 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.724 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.724 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.724 DEBUG:boto:Params: {} 2024-03-04T16:16:57.724 DEBUG:boto:Token: None 2024-03-04T16:16:57.724 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o 2024-03-04T16:16:57.724 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDT6esYdDa7vgUo5QRJ6fR7KMBo= 2024-03-04T16:16:57.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDT6esYdDa7vgUo5QRJ6fR7KMBo=', 'Content-Length': '0'} 2024-03-04T16:16:57.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003315488a28464d95-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.768 DEBUG:boto:path=/sbnvju-11/.o 2024-03-04T16:16:57.768 DEBUG:boto:auth_path=/sbnvju-11/.o 2024-03-04T16:16:57.768 DEBUG:boto:Method: GET 2024-03-04T16:16:57.768 DEBUG:boto:Path: /sbnvju-11/.o 2024-03-04T16:16:57.768 DEBUG:boto:Data: 2024-03-04T16:16:57.768 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.769 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.769 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.769 DEBUG:boto:Params: {} 2024-03-04T16:16:57.769 DEBUG:boto:Token: None 2024-03-04T16:16:57.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o 2024-03-04T16:16:57.769 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDT6esYdDa7vgUo5QRJ6fR7KMBo= 2024-03-04T16:16:57.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDT6esYdDa7vgUo5QRJ6fR7KMBo=', 'Content-Length': '0'} 2024-03-04T16:16:57.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5380f88f22a4f43-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.771 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.771 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.772 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.772 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.772 DEBUG:boto:Method: GET 2024-03-04T16:16:57.772 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:57.772 DEBUG:boto:Data: 2024-03-04T16:16:57.772 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.772 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.772 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.772 DEBUG:boto:Params: {} 2024-03-04T16:16:57.772 DEBUG:boto:Token: None 2024-03-04T16:16:57.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.773 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd7277f2f67b6877-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.775 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.776 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.776 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.776 DEBUG:boto:path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.776 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o 2024-03-04T16:16:57.776 DEBUG:boto:Method: GET 2024-03-04T16:16:57.776 DEBUG:boto:Path: /sbnvju-11/?prefix=.o 2024-03-04T16:16:57.776 DEBUG:boto:Data: 2024-03-04T16:16:57.776 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.776 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.777 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.777 DEBUG:boto:Params: {} 2024-03-04T16:16:57.777 DEBUG:boto:Token: None 2024-03-04T16:16:57.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ee6d6136e7f1b0-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.780 DEBUG:boto:b'sbnvju-11.o1000false.o2024-03-04T16:13:01.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:57.780 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.780 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.780 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.780 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.780 DEBUG:boto:Method: GET 2024-03-04T16:16:57.780 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.781 DEBUG:boto:Data: 2024-03-04T16:16:57.781 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.781 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.781 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.781 DEBUG:boto:Params: {} 2024-03-04T16:16:57.781 DEBUG:boto:Token: None 2024-03-04T16:16:57.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.781 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78= 2024-03-04T16:16:57.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78=', 'Content-Length': '0'} 2024-03-04T16:16:57.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e31974adac2e65d-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.784 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.784 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.784 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.784 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.784 DEBUG:boto:Method: GET 2024-03-04T16:16:57.784 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.784 DEBUG:boto:Data: 2024-03-04T16:16:57.785 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.785 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.785 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.785 DEBUG:boto:Params: {} 2024-03-04T16:16:57.785 DEBUG:boto:Token: None 2024-03-04T16:16:57.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78= 2024-03-04T16:16:57.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78=', 'Content-Length': '0'} 2024-03-04T16:16:57.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc503be37dda8f34-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.787 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.787 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.787 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.787 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.788 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.788 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.788 DEBUG:boto:Data: 2024-03-04T16:16:57.788 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.788 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.788 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.788 DEBUG:boto:Params: {} 2024-03-04T16:16:57.788 DEBUG:boto:Token: None 2024-03-04T16:16:57.788 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.788 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y52qcdiERaoYj9g5UnAe2gR13xQ= 2024-03-04T16:16:57.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y52qcdiERaoYj9g5UnAe2gR13xQ=', 'Content-Length': '0'} 2024-03-04T16:16:57.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031e7cb7f0a238632-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.791 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.791 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.791 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.791 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.791 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.791 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.791 DEBUG:boto:Data: 2024-03-04T16:16:57.791 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.791 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.791 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.791 DEBUG:boto:Params: {} 2024-03-04T16:16:57.792 DEBUG:boto:Token: None 2024-03-04T16:16:57.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y52qcdiERaoYj9g5UnAe2gR13xQ= 2024-03-04T16:16:57.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y52qcdiERaoYj9g5UnAe2gR13xQ=', 'Content-Length': '0'} 2024-03-04T16:16:57.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015f2bc860ea368e5-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:57.794 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.794 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.794 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.794 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.794 DEBUG:boto:Method: GET 2024-03-04T16:16:57.794 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.795 DEBUG:boto:Data: 2024-03-04T16:16:57.795 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.795 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.795 DEBUG:boto:Params: {} 2024-03-04T16:16:57.795 DEBUG:boto:Token: None 2024-03-04T16:16:57.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78= 2024-03-04T16:16:57.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78=', 'Content-Length': '0'} 2024-03-04T16:16:57.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000726e725b98739e91-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.840 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.840 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.840 DEBUG:boto:path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.841 DEBUG:boto:auth_path=/sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.841 DEBUG:boto:Method: GET 2024-03-04T16:16:57.841 DEBUG:boto:Path: /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.841 DEBUG:boto:Data: 2024-03-04T16:16:57.841 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.841 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.841 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.841 DEBUG:boto:Params: {} 2024-03-04T16:16:57.841 DEBUG:boto:Token: None 2024-03-04T16:16:57.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o.?versionId=null 2024-03-04T16:16:57.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78= 2024-03-04T16:16:57.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S7T7hXjd8knemN+tkD+xt/Ynd78=', 'Content-Length': '0'} 2024-03-04T16:16:57.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c060f87b3b47a4d-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.844 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.844 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.844 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.845 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.845 DEBUG:boto:Method: GET 2024-03-04T16:16:57.845 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.845 DEBUG:boto:Data: 2024-03-04T16:16:57.845 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.845 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.845 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.845 DEBUG:boto:Params: {} 2024-03-04T16:16:57.845 DEBUG:boto:Token: None 2024-03-04T16:16:57.845 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a8edd10a896780-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.892 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.892 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.892 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.893 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.893 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.893 DEBUG:boto:Method: GET 2024-03-04T16:16:57.893 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.893 DEBUG:boto:Data: 2024-03-04T16:16:57.893 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.893 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.893 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.893 DEBUG:boto:Params: {} 2024-03-04T16:16:57.893 DEBUG:boto:Token: None 2024-03-04T16:16:57.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.894 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b935d504174e8a-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.896 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.897 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.897 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.897 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.897 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:57.897 DEBUG:boto:Data: 2024-03-04T16:16:57.897 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.897 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.897 DEBUG:boto:Params: {} 2024-03-04T16:16:57.897 DEBUG:boto:Token: None 2024-03-04T16:16:57.898 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o. 2024-03-04T16:16:57.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gNYbadxqnAPjvTRdnWSrNo4fr6o= 2024-03-04T16:16:57.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gNYbadxqnAPjvTRdnWSrNo4fr6o=', 'Content-Length': '0'} 2024-03-04T16:16:57.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adebf1e182fd0f45-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.900 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.900 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.900 DEBUG:boto:Method: HEAD 2024-03-04T16:16:57.900 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:57.900 DEBUG:boto:Data: 2024-03-04T16:16:57.901 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.901 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.901 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.901 DEBUG:boto:Params: {} 2024-03-04T16:16:57.901 DEBUG:boto:Token: None 2024-03-04T16:16:57.901 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o. 2024-03-04T16:16:57.901 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gNYbadxqnAPjvTRdnWSrNo4fr6o= 2024-03-04T16:16:57.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gNYbadxqnAPjvTRdnWSrNo4fr6o=', 'Content-Length': '0'} 2024-03-04T16:16:57.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bf5bb2473cd7d08-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:57.903 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.903 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.903 DEBUG:boto:Method: GET 2024-03-04T16:16:57.903 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:57.903 DEBUG:boto:Data: 2024-03-04T16:16:57.904 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.904 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.904 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.904 DEBUG:boto:Params: {} 2024-03-04T16:16:57.904 DEBUG:boto:Token: None 2024-03-04T16:16:57.904 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o. 2024-03-04T16:16:57.904 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0sdUrvf2IDc/EByomr8QyeTmQSA= 2024-03-04T16:16:57.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0sdUrvf2IDc/EByomr8QyeTmQSA=', 'Content-Length': '0'} 2024-03-04T16:16:57.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004539b5688a84374f-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.948 DEBUG:boto:path=/sbnvju-11/.o. 2024-03-04T16:16:57.948 DEBUG:boto:auth_path=/sbnvju-11/.o. 2024-03-04T16:16:57.948 DEBUG:boto:Method: GET 2024-03-04T16:16:57.948 DEBUG:boto:Path: /sbnvju-11/.o. 2024-03-04T16:16:57.949 DEBUG:boto:Data: 2024-03-04T16:16:57.949 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.949 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.949 DEBUG:boto:Params: {} 2024-03-04T16:16:57.949 DEBUG:boto:Token: None 2024-03-04T16:16:57.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/.o. 2024-03-04T16:16:57.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0sdUrvf2IDc/EByomr8QyeTmQSA= 2024-03-04T16:16:57.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0sdUrvf2IDc/EByomr8QyeTmQSA=', 'Content-Length': '0'} 2024-03-04T16:16:57.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d998fbd82f1d4e0-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.952 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.952 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.952 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.952 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.952 DEBUG:boto:Method: GET 2024-03-04T16:16:57.952 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.952 DEBUG:boto:Data: 2024-03-04T16:16:57.952 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.952 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.953 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.953 DEBUG:boto:Params: {} 2024-03-04T16:16:57.953 DEBUG:boto:Token: None 2024-03-04T16:16:57.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.953 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000148be409862431c-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.956 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.956 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:57.957 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:57.957 DEBUG:boto:path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.957 DEBUG:boto:auth_path=/sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.957 DEBUG:boto:Method: GET 2024-03-04T16:16:57.957 DEBUG:boto:Path: /sbnvju-11/?prefix=.o. 2024-03-04T16:16:57.957 DEBUG:boto:Data: 2024-03-04T16:16:57.957 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.957 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:57.957 DEBUG:boto:Port: 8000 2024-03-04T16:16:57.957 DEBUG:boto:Params: {} 2024-03-04T16:16:57.957 DEBUG:boto:Token: None 2024-03-04T16:16:57.958 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/ 2024-03-04T16:16:57.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE= 2024-03-04T16:16:57.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nVpPwdc81E13sse6vrNu/2O8eyE=', 'Content-Length': '0'} 2024-03-04T16:16:57.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c364c2878aaeea94-0065e5f3f9-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:57.960 DEBUG:boto:b'sbnvju-11.o.1000false.o.2024-03-04T16:13:01.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:57.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:57.961 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:57.961 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:57.961 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:57.961 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:57.961 DEBUG:boto:Method: GET 2024-03-04T16:16:57.961 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:57.961 DEBUG:boto:Data: 2024-03-04T16:16:57.961 DEBUG:boto:Headers: {} 2024-03-04T16:16:57.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:57.962 DEBUG:boto:Port: 8001 2024-03-04T16:16:57.962 DEBUG:boto:Params: {} 2024-03-04T16:16:57.962 DEBUG:boto:Token: None 2024-03-04T16:16:57.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:57 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:57.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Dz6rhvGKrVEaUPgjl/SNsrFrgUQ= 2024-03-04T16:16:57.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Dz6rhvGKrVEaUPgjl/SNsrFrgUQ=', 'Content-Length': '0'} 2024-03-04T16:16:57.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f30fafff499c69cf-0065e5f3f9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.008 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.008 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.008 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.008 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.009 DEBUG:boto:Method: GET 2024-03-04T16:16:58.009 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.009 DEBUG:boto:Data: 2024-03-04T16:16:58.009 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.009 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.009 DEBUG:boto:Params: {} 2024-03-04T16:16:58.009 DEBUG:boto:Token: None 2024-03-04T16:16:58.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.009 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew= 2024-03-04T16:16:58.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew=', 'Content-Length': '0'} 2024-03-04T16:16:58.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fbc129dc9e5940e-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.012 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.012 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.012 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.012 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.012 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.012 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.013 DEBUG:boto:Data: 2024-03-04T16:16:58.013 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.013 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.013 DEBUG:boto:Params: {} 2024-03-04T16:16:58.013 DEBUG:boto:Token: None 2024-03-04T16:16:58.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3TfhyQtOk6/uvX8HOweD+3V0xq0= 2024-03-04T16:16:58.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3TfhyQtOk6/uvX8HOweD+3V0xq0=', 'Content-Length': '0'} 2024-03-04T16:16:58.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009def7986326b61ec-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.015 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.015 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.015 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.015 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.015 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.016 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.016 DEBUG:boto:Data: 2024-03-04T16:16:58.016 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.016 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.016 DEBUG:boto:Params: {} 2024-03-04T16:16:58.016 DEBUG:boto:Token: None 2024-03-04T16:16:58.016 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.016 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3TfhyQtOk6/uvX8HOweD+3V0xq0= 2024-03-04T16:16:58.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3TfhyQtOk6/uvX8HOweD+3V0xq0=', 'Content-Length': '0'} 2024-03-04T16:16:58.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f849b232fb816c8e-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:58.019 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.019 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.019 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.019 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.019 DEBUG:boto:Method: GET 2024-03-04T16:16:58.019 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.019 DEBUG:boto:Data: 2024-03-04T16:16:58.019 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.019 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.019 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.020 DEBUG:boto:Params: {} 2024-03-04T16:16:58.020 DEBUG:boto:Token: None 2024-03-04T16:16:58.020 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew= 2024-03-04T16:16:58.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew=', 'Content-Length': '0'} 2024-03-04T16:16:58.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c48e0a6580671e1-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.064 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.064 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.064 DEBUG:boto:path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.064 DEBUG:boto:auth_path=/sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.065 DEBUG:boto:Method: GET 2024-03-04T16:16:58.065 DEBUG:boto:Path: /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.065 DEBUG:boto:Data: 2024-03-04T16:16:58.065 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.065 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.065 DEBUG:boto:Params: {} 2024-03-04T16:16:58.065 DEBUG:boto:Token: None 2024-03-04T16:16:58.066 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A?versionId=null 2024-03-04T16:16:58.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew= 2024-03-04T16:16:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vsgnIseeWCY/fzx+NeYI7+5X9Ew=', 'Content-Length': '0'} 2024-03-04T16:16:58.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0b1f14081e776c2-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.068 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.068 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.068 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.068 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.068 DEBUG:boto:Method: GET 2024-03-04T16:16:58.068 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.069 DEBUG:boto:Data: 2024-03-04T16:16:58.069 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.069 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.069 DEBUG:boto:Params: {} 2024-03-04T16:16:58.069 DEBUG:boto:Token: None 2024-03-04T16:16:58.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000711bf413fffc860b-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.072 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.073 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.073 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.073 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.073 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.073 DEBUG:boto:Method: GET 2024-03-04T16:16:58.073 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.073 DEBUG:boto:Data: 2024-03-04T16:16:58.073 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.073 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.073 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.074 DEBUG:boto:Params: {} 2024-03-04T16:16:58.074 DEBUG:boto:Token: None 2024-03-04T16:16:58.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1c9305f57ff5c9-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.077 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.077 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.077 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.077 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.077 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:58.077 DEBUG:boto:Data: 2024-03-04T16:16:58.078 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.078 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.078 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.078 DEBUG:boto:Params: {} 2024-03-04T16:16:58.078 DEBUG:boto:Token: None 2024-03-04T16:16:58.078 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A 2024-03-04T16:16:58.078 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4NIcZOHEhFBIcvJ/6v2BdPP5fVQ= 2024-03-04T16:16:58.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4NIcZOHEhFBIcvJ/6v2BdPP5fVQ=', 'Content-Length': '0'} 2024-03-04T16:16:58.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095141779dfd49bb9-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.080 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.080 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.081 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.081 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:58.081 DEBUG:boto:Data: 2024-03-04T16:16:58.081 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.081 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.081 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.081 DEBUG:boto:Params: {} 2024-03-04T16:16:58.081 DEBUG:boto:Token: None 2024-03-04T16:16:58.081 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A 2024-03-04T16:16:58.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4NIcZOHEhFBIcvJ/6v2BdPP5fVQ= 2024-03-04T16:16:58.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4NIcZOHEhFBIcvJ/6v2BdPP5fVQ=', 'Content-Length': '0'} 2024-03-04T16:16:58.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075fae5f47e34576f-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:58.084 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.084 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.084 DEBUG:boto:Method: GET 2024-03-04T16:16:58.085 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:58.085 DEBUG:boto:Data: 2024-03-04T16:16:58.085 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.085 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.085 DEBUG:boto:Params: {} 2024-03-04T16:16:58.085 DEBUG:boto:Token: None 2024-03-04T16:16:58.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A 2024-03-04T16:16:58.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e9YIgHMfoS1pxy1+sx8HkVnT0kI= 2024-03-04T16:16:58.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e9YIgHMfoS1pxy1+sx8HkVnT0kI=', 'Content-Length': '0'} 2024-03-04T16:16:58.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059e0b5cdf7fe5a55-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.128 DEBUG:boto:path=/sbnvju-11/%3A 2024-03-04T16:16:58.128 DEBUG:boto:auth_path=/sbnvju-11/%3A 2024-03-04T16:16:58.128 DEBUG:boto:Method: GET 2024-03-04T16:16:58.128 DEBUG:boto:Path: /sbnvju-11/%3A 2024-03-04T16:16:58.128 DEBUG:boto:Data: 2024-03-04T16:16:58.129 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.129 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.129 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.129 DEBUG:boto:Params: {} 2024-03-04T16:16:58.129 DEBUG:boto:Token: None 2024-03-04T16:16:58.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/%3A 2024-03-04T16:16:58.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e9YIgHMfoS1pxy1+sx8HkVnT0kI= 2024-03-04T16:16:58.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e9YIgHMfoS1pxy1+sx8HkVnT0kI=', 'Content-Length': '0'} 2024-03-04T16:16:58.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000217dd19b10a18d74-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.131 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.132 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.132 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.132 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.132 DEBUG:boto:Method: GET 2024-03-04T16:16:58.132 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.132 DEBUG:boto:Data: 2024-03-04T16:16:58.132 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.132 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.132 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.132 DEBUG:boto:Params: {} 2024-03-04T16:16:58.133 DEBUG:boto:Token: None 2024-03-04T16:16:58.133 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90d7f4d460dedd9-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.136 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.136 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.136 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.136 DEBUG:boto:path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.136 DEBUG:boto:auth_path=/sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.136 DEBUG:boto:Method: GET 2024-03-04T16:16:58.137 DEBUG:boto:Path: /sbnvju-11/?prefix=%3A 2024-03-04T16:16:58.137 DEBUG:boto:Data: 2024-03-04T16:16:58.137 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.137 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.137 DEBUG:boto:Params: {} 2024-03-04T16:16:58.137 DEBUG:boto:Token: None 2024-03-04T16:16:58.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078df3f0eadceab90-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.140 DEBUG:boto:b'sbnvju-11:1000false:2024-03-04T16:13:01.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:58.140 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.140 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.140 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.141 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.141 DEBUG:boto:Method: GET 2024-03-04T16:16:58.141 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.141 DEBUG:boto:Data: 2024-03-04T16:16:58.141 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.141 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.141 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.141 DEBUG:boto:Params: {} 2024-03-04T16:16:58.141 DEBUG:boto:Token: None 2024-03-04T16:16:58.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg= 2024-03-04T16:16:58.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000023bee48d0423e65-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.184 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.184 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.184 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.185 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.185 DEBUG:boto:Method: GET 2024-03-04T16:16:58.185 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.185 DEBUG:boto:Data: 2024-03-04T16:16:58.185 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.185 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.185 DEBUG:boto:Params: {} 2024-03-04T16:16:58.185 DEBUG:boto:Token: None 2024-03-04T16:16:58.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg= 2024-03-04T16:16:58.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061a8e45527ab2bfc-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.188 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.188 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.188 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.188 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.188 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.188 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.188 DEBUG:boto:Data: 2024-03-04T16:16:58.188 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.188 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.188 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.189 DEBUG:boto:Params: {} 2024-03-04T16:16:58.189 DEBUG:boto:Token: None 2024-03-04T16:16:58.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HjHDoyRiXQEm9LStmDofE1C+Fjc= 2024-03-04T16:16:58.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HjHDoyRiXQEm9LStmDofE1C+Fjc=', 'Content-Length': '0'} 2024-03-04T16:16:58.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000075b5b3be0181ae4-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.191 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.191 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.191 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.192 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.192 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.192 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.192 DEBUG:boto:Data: 2024-03-04T16:16:58.192 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.192 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.192 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.192 DEBUG:boto:Params: {} 2024-03-04T16:16:58.192 DEBUG:boto:Token: None 2024-03-04T16:16:58.192 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.192 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HjHDoyRiXQEm9LStmDofE1C+Fjc= 2024-03-04T16:16:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HjHDoyRiXQEm9LStmDofE1C+Fjc=', 'Content-Length': '0'} 2024-03-04T16:16:58.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c01f055368b4588b-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:58.195 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.195 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.195 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.195 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.195 DEBUG:boto:Method: GET 2024-03-04T16:16:58.195 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.196 DEBUG:boto:Data: 2024-03-04T16:16:58.196 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.196 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.196 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.196 DEBUG:boto:Params: {} 2024-03-04T16:16:58.196 DEBUG:boto:Token: None 2024-03-04T16:16:58.196 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.196 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg= 2024-03-04T16:16:58.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e87745e23b66cf8-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.240 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.240 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.240 DEBUG:boto:path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.240 DEBUG:boto:auth_path=/sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.240 DEBUG:boto:Method: GET 2024-03-04T16:16:58.240 DEBUG:boto:Path: /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.241 DEBUG:boto:Data: 2024-03-04T16:16:58.241 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.241 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.241 DEBUG:boto:Params: {} 2024-03-04T16:16:58.241 DEBUG:boto:Token: None 2024-03-04T16:16:58.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj?versionId=null 2024-03-04T16:16:58.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg= 2024-03-04T16:16:58.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xbhq4javTHEs/983HCVGgWmetkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047c7fac495225a86-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.243 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.243 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.243 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.244 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.244 DEBUG:boto:Method: GET 2024-03-04T16:16:58.244 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.244 DEBUG:boto:Data: 2024-03-04T16:16:58.244 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.244 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.244 DEBUG:boto:Params: {} 2024-03-04T16:16:58.244 DEBUG:boto:Token: None 2024-03-04T16:16:58.244 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad1d9a5a30a3fbd-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.288 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.288 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.289 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.289 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.289 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.289 DEBUG:boto:Method: GET 2024-03-04T16:16:58.289 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.289 DEBUG:boto:Data: 2024-03-04T16:16:58.289 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.289 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.289 DEBUG:boto:Params: {} 2024-03-04T16:16:58.289 DEBUG:boto:Token: None 2024-03-04T16:16:58.290 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049de37bd90c367a4-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.292 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.293 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.293 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.293 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.293 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:58.293 DEBUG:boto:Data: 2024-03-04T16:16:58.293 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.293 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.293 DEBUG:boto:Params: {} 2024-03-04T16:16:58.293 DEBUG:boto:Token: None 2024-03-04T16:16:58.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj 2024-03-04T16:16:58.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKAMhQ+fI+uw1o6pwfu371Rp1J0= 2024-03-04T16:16:58.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKAMhQ+fI+uw1o6pwfu371Rp1J0=', 'Content-Length': '0'} 2024-03-04T16:16:58.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f7b45864e36a061-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.296 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.296 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.296 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.296 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:58.296 DEBUG:boto:Data: 2024-03-04T16:16:58.296 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.296 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.296 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.296 DEBUG:boto:Params: {} 2024-03-04T16:16:58.297 DEBUG:boto:Token: None 2024-03-04T16:16:58.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj 2024-03-04T16:16:58.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKAMhQ+fI+uw1o6pwfu371Rp1J0= 2024-03-04T16:16:58.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKAMhQ+fI+uw1o6pwfu371Rp1J0=', 'Content-Length': '0'} 2024-03-04T16:16:58.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d56ad20d5a610bcd-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:16:58.299 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.299 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.299 DEBUG:boto:Method: GET 2024-03-04T16:16:58.299 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:58.300 DEBUG:boto:Data: 2024-03-04T16:16:58.300 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.300 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.300 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.300 DEBUG:boto:Params: {} 2024-03-04T16:16:58.300 DEBUG:boto:Token: None 2024-03-04T16:16:58.300 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj 2024-03-04T16:16:58.300 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+rXbFaLykGc3pRsklgZJYtMoIfU= 2024-03-04T16:16:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+rXbFaLykGc3pRsklgZJYtMoIfU=', 'Content-Length': '0'} 2024-03-04T16:16:58.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000221b2e20c4769d2c-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.344 DEBUG:boto:path=/sbnvju-11/_myobj 2024-03-04T16:16:58.344 DEBUG:boto:auth_path=/sbnvju-11/_myobj 2024-03-04T16:16:58.344 DEBUG:boto:Method: GET 2024-03-04T16:16:58.344 DEBUG:boto:Path: /sbnvju-11/_myobj 2024-03-04T16:16:58.344 DEBUG:boto:Data: 2024-03-04T16:16:58.345 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.345 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.345 DEBUG:boto:Params: {} 2024-03-04T16:16:58.345 DEBUG:boto:Token: None 2024-03-04T16:16:58.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/_myobj 2024-03-04T16:16:58.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+rXbFaLykGc3pRsklgZJYtMoIfU= 2024-03-04T16:16:58.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+rXbFaLykGc3pRsklgZJYtMoIfU=', 'Content-Length': '0'} 2024-03-04T16:16:58.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000241ebf152caba36a-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.347 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.347 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.347 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.348 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.348 DEBUG:boto:Method: GET 2024-03-04T16:16:58.348 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.348 DEBUG:boto:Data: 2024-03-04T16:16:58.348 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.348 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.348 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.348 DEBUG:boto:Params: {} 2024-03-04T16:16:58.348 DEBUG:boto:Token: None 2024-03-04T16:16:58.348 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb5d791e5e0361e0-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.351 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.351 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.352 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.352 DEBUG:boto:path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.352 DEBUG:boto:auth_path=/sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.352 DEBUG:boto:Method: GET 2024-03-04T16:16:58.352 DEBUG:boto:Path: /sbnvju-11/?prefix=_myobj 2024-03-04T16:16:58.352 DEBUG:boto:Data: 2024-03-04T16:16:58.352 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.352 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.352 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.352 DEBUG:boto:Params: {} 2024-03-04T16:16:58.352 DEBUG:boto:Token: None 2024-03-04T16:16:58.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2be295a5a2ae44-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.355 DEBUG:boto:b'sbnvju-11_myobj1000false_myobj2024-03-04T16:13:01.352Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:58.356 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.356 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.356 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.356 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.356 DEBUG:boto:Method: GET 2024-03-04T16:16:58.356 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.356 DEBUG:boto:Data: 2024-03-04T16:16:58.356 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.357 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.357 DEBUG:boto:Params: {} 2024-03-04T16:16:58.357 DEBUG:boto:Token: None 2024-03-04T16:16:58.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI= 2024-03-04T16:16:58.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI=', 'Content-Length': '0'} 2024-03-04T16:16:58.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1f364945ce0ca9-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.359 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.359 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.359 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.360 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.360 DEBUG:boto:Method: GET 2024-03-04T16:16:58.360 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.360 DEBUG:boto:Data: 2024-03-04T16:16:58.360 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.360 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.360 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.360 DEBUG:boto:Params: {} 2024-03-04T16:16:58.360 DEBUG:boto:Token: None 2024-03-04T16:16:58.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI= 2024-03-04T16:16:58.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI=', 'Content-Length': '0'} 2024-03-04T16:16:58.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd63e95fdf516bcc-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.363 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.363 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.363 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.363 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.363 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.363 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.363 DEBUG:boto:Data: 2024-03-04T16:16:58.364 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.364 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.364 DEBUG:boto:Params: {} 2024-03-04T16:16:58.364 DEBUG:boto:Token: None 2024-03-04T16:16:58.364 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.364 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N3LrNU3+vjpDZ1ixSXbH2xWwUbQ= 2024-03-04T16:16:58.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N3LrNU3+vjpDZ1ixSXbH2xWwUbQ=', 'Content-Length': '0'} 2024-03-04T16:16:58.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000102a9446f62f83c1-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.367 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.367 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.367 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.367 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.367 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.367 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.367 DEBUG:boto:Data: 2024-03-04T16:16:58.367 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.367 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.367 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.367 DEBUG:boto:Params: {} 2024-03-04T16:16:58.368 DEBUG:boto:Token: None 2024-03-04T16:16:58.368 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.368 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N3LrNU3+vjpDZ1ixSXbH2xWwUbQ= 2024-03-04T16:16:58.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N3LrNU3+vjpDZ1ixSXbH2xWwUbQ=', 'Content-Length': '0'} 2024-03-04T16:16:58.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c98e444690132f1c-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:58.370 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.370 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.371 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.371 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.371 DEBUG:boto:Method: GET 2024-03-04T16:16:58.371 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.371 DEBUG:boto:Data: 2024-03-04T16:16:58.371 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.371 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.371 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.371 DEBUG:boto:Params: {} 2024-03-04T16:16:58.372 DEBUG:boto:Token: None 2024-03-04T16:16:58.372 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.372 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI= 2024-03-04T16:16:58.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI=', 'Content-Length': '0'} 2024-03-04T16:16:58.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd5ba1219ff4cfd8-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.416 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.416 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.416 DEBUG:boto:path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.416 DEBUG:boto:auth_path=/sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.417 DEBUG:boto:Method: GET 2024-03-04T16:16:58.417 DEBUG:boto:Path: /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.417 DEBUG:boto:Data: 2024-03-04T16:16:58.417 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.417 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.417 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.417 DEBUG:boto:Params: {} 2024-03-04T16:16:58.417 DEBUG:boto:Token: None 2024-03-04T16:16:58.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj?versionId=null 2024-03-04T16:16:58.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI= 2024-03-04T16:16:58.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:++MfRUQ5KmjtCrfDQHQ57+IzedI=', 'Content-Length': '0'} 2024-03-04T16:16:58.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073457d96615c0f7c-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.420 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.420 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.420 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.420 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.420 DEBUG:boto:Method: GET 2024-03-04T16:16:58.420 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.420 DEBUG:boto:Data: 2024-03-04T16:16:58.420 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.420 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.421 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.421 DEBUG:boto:Params: {} 2024-03-04T16:16:58.421 DEBUG:boto:Token: None 2024-03-04T16:16:58.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab0a2267a09e5d6-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.464 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.465 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.465 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.465 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.465 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.465 DEBUG:boto:Method: GET 2024-03-04T16:16:58.465 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.465 DEBUG:boto:Data: 2024-03-04T16:16:58.465 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.466 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.466 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.466 DEBUG:boto:Params: {} 2024-03-04T16:16:58.466 DEBUG:boto:Token: None 2024-03-04T16:16:58.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005373167083c16a36-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.469 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.469 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.469 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.470 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.470 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:58.470 DEBUG:boto:Data: 2024-03-04T16:16:58.470 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.470 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.470 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.470 DEBUG:boto:Params: {} 2024-03-04T16:16:58.470 DEBUG:boto:Token: None 2024-03-04T16:16:58.470 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj 2024-03-04T16:16:58.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a43s2mBMVecA/WikB4Bz/2mfMC4= 2024-03-04T16:16:58.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a43s2mBMVecA/WikB4Bz/2mfMC4=', 'Content-Length': '0'} 2024-03-04T16:16:58.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aadda46c4f3787ab-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.472 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.472 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.472 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.473 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:58.473 DEBUG:boto:Data: 2024-03-04T16:16:58.473 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.473 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.473 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.473 DEBUG:boto:Params: {} 2024-03-04T16:16:58.473 DEBUG:boto:Token: None 2024-03-04T16:16:58.473 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj 2024-03-04T16:16:58.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a43s2mBMVecA/WikB4Bz/2mfMC4= 2024-03-04T16:16:58.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a43s2mBMVecA/WikB4Bz/2mfMC4=', 'Content-Length': '0'} 2024-03-04T16:16:58.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cad8fc492bd5a8a8-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:16:58.475 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.476 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.476 DEBUG:boto:Method: GET 2024-03-04T16:16:58.476 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:58.476 DEBUG:boto:Data: 2024-03-04T16:16:58.476 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.476 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.476 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.476 DEBUG:boto:Params: {} 2024-03-04T16:16:58.476 DEBUG:boto:Token: None 2024-03-04T16:16:58.476 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj 2024-03-04T16:16:58.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hEaLkSxC6c31z7pGXBmUgaXUu2g= 2024-03-04T16:16:58.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hEaLkSxC6c31z7pGXBmUgaXUu2g=', 'Content-Length': '0'} 2024-03-04T16:16:58.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-11:fc294220-5038-4f25-aeee-cac2c5838729.4341.20'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f0476b1a009820c-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.520 DEBUG:boto:path=/sbnvju-11/myobj 2024-03-04T16:16:58.520 DEBUG:boto:auth_path=/sbnvju-11/myobj 2024-03-04T16:16:58.520 DEBUG:boto:Method: GET 2024-03-04T16:16:58.520 DEBUG:boto:Path: /sbnvju-11/myobj 2024-03-04T16:16:58.520 DEBUG:boto:Data: 2024-03-04T16:16:58.520 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.521 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.521 DEBUG:boto:Params: {} 2024-03-04T16:16:58.521 DEBUG:boto:Token: None 2024-03-04T16:16:58.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/myobj 2024-03-04T16:16:58.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hEaLkSxC6c31z7pGXBmUgaXUu2g= 2024-03-04T16:16:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hEaLkSxC6c31z7pGXBmUgaXUu2g=', 'Content-Length': '0'} 2024-03-04T16:16:58.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fba4aa6880d08ff6-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.523 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.524 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.524 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.524 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.524 DEBUG:boto:Method: GET 2024-03-04T16:16:58.524 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.524 DEBUG:boto:Data: 2024-03-04T16:16:58.524 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.524 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.524 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.524 DEBUG:boto:Params: {} 2024-03-04T16:16:58.525 DEBUG:boto:Token: None 2024-03-04T16:16:58.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ec4fdc032d6433-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.568 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.568 DEBUG:boto:path=/sbnvju-11/ 2024-03-04T16:16:58.568 DEBUG:boto:auth_path=/sbnvju-11/ 2024-03-04T16:16:58.569 DEBUG:boto:path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.569 DEBUG:boto:auth_path=/sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.569 DEBUG:boto:Method: GET 2024-03-04T16:16:58.569 DEBUG:boto:Path: /sbnvju-11/?prefix=myobj 2024-03-04T16:16:58.569 DEBUG:boto:Data: 2024-03-04T16:16:58.569 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.569 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:16:58.569 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.569 DEBUG:boto:Params: {} 2024-03-04T16:16:58.569 DEBUG:boto:Token: None 2024-03-04T16:16:58.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-11/ 2024-03-04T16:16:58.569 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg= 2024-03-04T16:16:58.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YmJkEmW32z6dIH4w+jVrGap4bkg=', 'Content-Length': '0'} 2024-03-04T16:16:58.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cf773d8e004aa4-0065e5f3fa-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.573 DEBUG:boto:b'sbnvju-11myobj1000falsemyobj2024-03-04T16:13:01.339Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.573 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-11 zones={test-zone3, test-zone2} 2024-03-04T16:16:58.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:16:58.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:16:58.574 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:16:58.636 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:16:58.636 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:16:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:58.693+0000 7f699562e800 1 bucket sync caught up with source: 2024-03-04T16:16:58.694 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.12.6, , 00000000001.15.6, , 00000000001.17.6, 00000000002.17.6, 00000000001.9.6, 00000000001.14.6, 00000000001.13.6, ] 2024-03-04T16:16:58.694 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.12.6, , 00000000001.15.6, , 00000000001.17.6, 00000000002.17.6, 00000000001.9.6, 00000000001.14.6, 00000000001.13.6, ] 2024-03-04T16:16:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:16:58.693+0000 7f699562e800 0 bucket checkpoint complete 2024-03-04T16:16:58.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-12 zones={test-zone1, test-zone3} 2024-03-04T16:16:58.714 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.714 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.714 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.714 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:16:58.714 DEBUG:boto:Data: 2024-03-04T16:16:58.714 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.714 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.714 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.714 DEBUG:boto:Params: {} 2024-03-04T16:16:58.715 DEBUG:boto:Token: None 2024-03-04T16:16:58.715 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.715 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:br3XhNOarzTTfgFmsYOAEGKbnjY= 2024-03-04T16:16:58.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:br3XhNOarzTTfgFmsYOAEGKbnjY=', 'Content-Length': '0'} 2024-03-04T16:16:58.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9d19f545b240de6-0065e5f3fa-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.717 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.718 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.718 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.718 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:16:58.718 DEBUG:boto:Data: 2024-03-04T16:16:58.718 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.718 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.718 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.718 DEBUG:boto:Params: {} 2024-03-04T16:16:58.718 DEBUG:boto:Token: None 2024-03-04T16:16:58.718 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.719 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:br3XhNOarzTTfgFmsYOAEGKbnjY= 2024-03-04T16:16:58.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:br3XhNOarzTTfgFmsYOAEGKbnjY=', 'Content-Length': '0'} 2024-03-04T16:16:58.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066218758adf51a15-0065e5f3fa-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:16:58.721 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.721 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.722 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.722 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.722 DEBUG:boto:Method: GET 2024-03-04T16:16:58.722 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.722 DEBUG:boto:Data: 2024-03-04T16:16:58.722 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.722 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.722 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.722 DEBUG:boto:Params: {} 2024-03-04T16:16:58.722 DEBUG:boto:Token: None 2024-03-04T16:16:58.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/?versions 2024-03-04T16:16:58.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A= 2024-03-04T16:16:58.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A=', 'Content-Length': '0'} 2024-03-04T16:16:58.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000651b894a513a08e-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.725 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.725 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:58.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:16:58.726 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.726 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.726 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.727 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.727 DEBUG:boto:Method: GET 2024-03-04T16:16:58.727 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.727 DEBUG:boto:Data: 2024-03-04T16:16:58.727 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.727 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.727 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.727 DEBUG:boto:Params: {} 2024-03-04T16:16:58.727 DEBUG:boto:Token: None 2024-03-04T16:16:58.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/?versions 2024-03-04T16:16:58.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A= 2024-03-04T16:16:58.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A=', 'Content-Length': '0'} 2024-03-04T16:16:58.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe9a0b57ec8b69d-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.730 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:16:58.731 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:16:58.731 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.732 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.732 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.732 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.732 DEBUG:boto:Method: GET 2024-03-04T16:16:58.732 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.732 DEBUG:boto:Data: 2024-03-04T16:16:58.732 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.732 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.732 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.732 DEBUG:boto:Params: {} 2024-03-04T16:16:58.733 DEBUG:boto:Token: None 2024-03-04T16:16:58.733 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/?versions 2024-03-04T16:16:58.733 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A= 2024-03-04T16:16:58.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A=', 'Content-Length': '0'} 2024-03-04T16:16:58.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a15e1b1fcd318ec-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.735 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.736 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.736 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.736 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.736 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.736 DEBUG:boto:Method: GET 2024-03-04T16:16:58.737 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:16:58.737 DEBUG:boto:Data: 2024-03-04T16:16:58.737 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.737 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.737 DEBUG:boto:Params: {} 2024-03-04T16:16:58.737 DEBUG:boto:Token: None 2024-03-04T16:16:58.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/?versions 2024-03-04T16:16:58.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A= 2024-03-04T16:16:58.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gL+/SPcf42nf6VdR6hjx0Cghi2A=', 'Content-Length': '0'} 2024-03-04T16:16:58.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557d4c5f544e5064-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.740 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:58.741 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.741 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.741 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.741 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.741 DEBUG:boto:Method: GET 2024-03-04T16:16:58.741 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.741 DEBUG:boto:Data: 2024-03-04T16:16:58.741 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.742 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.742 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.742 DEBUG:boto:Params: {} 2024-03-04T16:16:58.742 DEBUG:boto:Token: None 2024-03-04T16:16:58.742 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.742 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs= 2024-03-04T16:16:58.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs=', 'Content-Length': '0'} 2024-03-04T16:16:58.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004613794c38b44271-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.784 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.784 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.784 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.784 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.784 DEBUG:boto:Method: GET 2024-03-04T16:16:58.785 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.785 DEBUG:boto:Data: 2024-03-04T16:16:58.785 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.785 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.785 DEBUG:boto:Params: {} 2024-03-04T16:16:58.785 DEBUG:boto:Token: None 2024-03-04T16:16:58.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs= 2024-03-04T16:16:58.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs=', 'Content-Length': '0'} 2024-03-04T16:16:58.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b54eb600afe5e674-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.787 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.787 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.787 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.788 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.788 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.788 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.788 DEBUG:boto:Data: 2024-03-04T16:16:58.788 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.788 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.788 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.788 DEBUG:boto:Params: {} 2024-03-04T16:16:58.788 DEBUG:boto:Token: None 2024-03-04T16:16:58.788 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.788 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WXMiQl+636I0lg7C2ROnE9PFdPc= 2024-03-04T16:16:58.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WXMiQl+636I0lg7C2ROnE9PFdPc=', 'Content-Length': '0'} 2024-03-04T16:16:58.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b5bc9a860bae883-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.790 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.790 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.791 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.791 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.791 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.791 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.791 DEBUG:boto:Data: 2024-03-04T16:16:58.791 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.791 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.791 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.791 DEBUG:boto:Params: {} 2024-03-04T16:16:58.791 DEBUG:boto:Token: None 2024-03-04T16:16:58.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WXMiQl+636I0lg7C2ROnE9PFdPc= 2024-03-04T16:16:58.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WXMiQl+636I0lg7C2ROnE9PFdPc=', 'Content-Length': '0'} 2024-03-04T16:16:58.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7ea9a6936d86f6e-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:58.794 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.794 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.794 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.794 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.794 DEBUG:boto:Method: GET 2024-03-04T16:16:58.794 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.794 DEBUG:boto:Data: 2024-03-04T16:16:58.794 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.794 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.794 DEBUG:boto:Params: {} 2024-03-04T16:16:58.795 DEBUG:boto:Token: None 2024-03-04T16:16:58.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs= 2024-03-04T16:16:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs=', 'Content-Length': '0'} 2024-03-04T16:16:58.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0c6a3141ee2187a-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.797 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.797 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.797 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.797 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.797 DEBUG:boto:Method: GET 2024-03-04T16:16:58.797 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.797 DEBUG:boto:Data: 2024-03-04T16:16:58.797 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.797 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.797 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.797 DEBUG:boto:Params: {} 2024-03-04T16:16:58.798 DEBUG:boto:Token: None 2024-03-04T16:16:58.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:16:58.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs= 2024-03-04T16:16:58.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sfcfSC8WDYIHf4naSm3uuyeNOVs=', 'Content-Length': '0'} 2024-03-04T16:16:58.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c730d91c133b72e6-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.800 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.800 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.800 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.800 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.800 DEBUG:boto:Method: GET 2024-03-04T16:16:58.800 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:16:58.800 DEBUG:boto:Data: 2024-03-04T16:16:58.800 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.800 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.800 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.801 DEBUG:boto:Params: {} 2024-03-04T16:16:58.801 DEBUG:boto:Token: None 2024-03-04T16:16:58.801 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.801 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw= 2024-03-04T16:16:58.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw=', 'Content-Length': '0'} 2024-03-04T16:16:58.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d34f7a7ac26b09-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.803 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.804 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.804 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.804 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.804 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.804 DEBUG:boto:Method: GET 2024-03-04T16:16:58.804 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:16:58.805 DEBUG:boto:Data: 2024-03-04T16:16:58.805 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.805 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.805 DEBUG:boto:Params: {} 2024-03-04T16:16:58.805 DEBUG:boto:Token: None 2024-03-04T16:16:58.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw= 2024-03-04T16:16:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw=', 'Content-Length': '0'} 2024-03-04T16:16:58.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c130bdbbc9312159-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.808 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.808 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.808 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.808 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.808 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:16:58.809 DEBUG:boto:Data: 2024-03-04T16:16:58.809 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.809 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.809 DEBUG:boto:Params: {} 2024-03-04T16:16:58.809 DEBUG:boto:Token: None 2024-03-04T16:16:58.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26 2024-03-04T16:16:58.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sjxWloVxwJfAuRzuVcZnhfDkwY8= 2024-03-04T16:16:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sjxWloVxwJfAuRzuVcZnhfDkwY8=', 'Content-Length': '0'} 2024-03-04T16:16:58.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a76f234645433a7-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.811 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.811 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.811 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.811 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:16:58.812 DEBUG:boto:Data: 2024-03-04T16:16:58.812 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.812 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.812 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.812 DEBUG:boto:Params: {} 2024-03-04T16:16:58.812 DEBUG:boto:Token: None 2024-03-04T16:16:58.812 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26 2024-03-04T16:16:58.812 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sjxWloVxwJfAuRzuVcZnhfDkwY8= 2024-03-04T16:16:58.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sjxWloVxwJfAuRzuVcZnhfDkwY8=', 'Content-Length': '0'} 2024-03-04T16:16:58.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f75bf5108f001821-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:16:58.814 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.814 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.815 DEBUG:boto:Method: GET 2024-03-04T16:16:58.815 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:16:58.815 DEBUG:boto:Data: 2024-03-04T16:16:58.815 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.815 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.815 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.815 DEBUG:boto:Params: {} 2024-03-04T16:16:58.815 DEBUG:boto:Token: None 2024-03-04T16:16:58.815 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26 2024-03-04T16:16:58.815 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mp5+Vv5FnSXBLer0z7DWlcF67ew= 2024-03-04T16:16:58.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mp5+Vv5FnSXBLer0z7DWlcF67ew=', 'Content-Length': '0'} 2024-03-04T16:16:58.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dccc08ea3ba9f9e-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.860 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:16:58.860 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:16:58.860 DEBUG:boto:Method: GET 2024-03-04T16:16:58.860 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:16:58.861 DEBUG:boto:Data: 2024-03-04T16:16:58.861 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.861 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.861 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.861 DEBUG:boto:Params: {} 2024-03-04T16:16:58.861 DEBUG:boto:Token: None 2024-03-04T16:16:58.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/%26 2024-03-04T16:16:58.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Mp5+Vv5FnSXBLer0z7DWlcF67ew= 2024-03-04T16:16:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Mp5+Vv5FnSXBLer0z7DWlcF67ew=', 'Content-Length': '0'} 2024-03-04T16:16:58.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f782d9b58158c71-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.863 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.863 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.863 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.863 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.864 DEBUG:boto:Method: GET 2024-03-04T16:16:58.864 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:16:58.864 DEBUG:boto:Data: 2024-03-04T16:16:58.864 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.864 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.864 DEBUG:boto:Params: {} 2024-03-04T16:16:58.864 DEBUG:boto:Token: None 2024-03-04T16:16:58.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw= 2024-03-04T16:16:58.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw=', 'Content-Length': '0'} 2024-03-04T16:16:58.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea07f5dfb956004-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.908 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.908 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:58.908 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:58.909 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.909 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:16:58.909 DEBUG:boto:Method: GET 2024-03-04T16:16:58.909 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:16:58.909 DEBUG:boto:Data: 2024-03-04T16:16:58.909 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.909 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.909 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.909 DEBUG:boto:Params: {} 2024-03-04T16:16:58.909 DEBUG:boto:Token: None 2024-03-04T16:16:58.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/ 2024-03-04T16:16:58.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw= 2024-03-04T16:16:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pk7ZHUTc4ThZaimByuBpJD/atKw=', 'Content-Length': '0'} 2024-03-04T16:16:58.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17834754f9a0db1-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.912 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:58.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:58.913 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:58.913 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:58.913 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.913 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.913 DEBUG:boto:Method: GET 2024-03-04T16:16:58.913 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.913 DEBUG:boto:Data: 2024-03-04T16:16:58.913 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.913 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.914 DEBUG:boto:Params: {} 2024-03-04T16:16:58.914 DEBUG:boto:Token: None 2024-03-04T16:16:58.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY= 2024-03-04T16:16:58.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY=', 'Content-Length': '0'} 2024-03-04T16:16:58.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008af009b97e5f162d-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.956 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:58.956 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:58.956 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.956 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.956 DEBUG:boto:Method: GET 2024-03-04T16:16:58.956 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.957 DEBUG:boto:Data: 2024-03-04T16:16:58.957 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.957 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.957 DEBUG:boto:Params: {} 2024-03-04T16:16:58.957 DEBUG:boto:Token: None 2024-03-04T16:16:58.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY= 2024-03-04T16:16:58.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY=', 'Content-Length': '0'} 2024-03-04T16:16:58.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cefd536d7708f4b-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.960 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:58.960 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:58.960 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.960 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.960 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.960 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.960 DEBUG:boto:Data: 2024-03-04T16:16:58.960 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.961 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.961 DEBUG:boto:Params: {} 2024-03-04T16:16:58.961 DEBUG:boto:Token: None 2024-03-04T16:16:58.961 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.961 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MZkwUVaN85fpXoiXJYt0YmKuwEo= 2024-03-04T16:16:58.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MZkwUVaN85fpXoiXJYt0YmKuwEo=', 'Content-Length': '0'} 2024-03-04T16:16:58.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f84c08022e17690c-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.963 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:58.963 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:58.964 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.964 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.964 DEBUG:boto:Method: HEAD 2024-03-04T16:16:58.964 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.964 DEBUG:boto:Data: 2024-03-04T16:16:58.964 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.964 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:58.964 DEBUG:boto:Port: 8001 2024-03-04T16:16:58.964 DEBUG:boto:Params: {} 2024-03-04T16:16:58.964 DEBUG:boto:Token: None 2024-03-04T16:16:58.964 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.965 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MZkwUVaN85fpXoiXJYt0YmKuwEo= 2024-03-04T16:16:58.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MZkwUVaN85fpXoiXJYt0YmKuwEo=', 'Content-Length': '0'} 2024-03-04T16:16:58.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000340041792994909c-0065e5f3fa-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:58.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:58.966 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:58.966 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:58.966 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.967 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:58.967 DEBUG:boto:Method: GET 2024-03-04T16:16:58.967 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.967 DEBUG:boto:Data: 2024-03-04T16:16:58.967 DEBUG:boto:Headers: {} 2024-03-04T16:16:58.967 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:58.967 DEBUG:boto:Port: 8000 2024-03-04T16:16:58.967 DEBUG:boto:Params: {} 2024-03-04T16:16:58.967 DEBUG:boto:Token: None 2024-03-04T16:16:58.967 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:58 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:58.968 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY= 2024-03-04T16:16:58.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fAEWFOBZViQF6p4rqtGCJwFdVxY=', 'Content-Length': '0'} 2024-03-04T16:16:58.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000227f5e1825f8219e-0065e5f3fa-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.012 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:59.012 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:59.012 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:59.012 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:16:59.012 DEBUG:boto:Method: GET 2024-03-04T16:16:59.013 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:16:59.013 DEBUG:boto:Data: 2024-03-04T16:16:59.013 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.013 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.013 DEBUG:boto:Params: {} 2024-03-04T16:16:59.013 DEBUG:boto:Token: None 2024-03-04T16:16:59.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:16:59.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cYSdRKCwr7P+EV7+7nUndjnbASA= 2024-03-04T16:16:59.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cYSdRKCwr7P+EV7+7nUndjnbASA=', 'Content-Length': '0'} 2024-03-04T16:16:59.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f445882e32611b-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.016 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.016 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.016 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.016 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.016 DEBUG:boto:Method: GET 2024-03-04T16:16:59.016 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:16:59.016 DEBUG:boto:Data: 2024-03-04T16:16:59.016 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.016 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.016 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.016 DEBUG:boto:Params: {} 2024-03-04T16:16:59.017 DEBUG:boto:Token: None 2024-03-04T16:16:59.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce929ac33b95e9be-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.019 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.020 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.020 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.020 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.020 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.020 DEBUG:boto:Method: GET 2024-03-04T16:16:59.020 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:16:59.021 DEBUG:boto:Data: 2024-03-04T16:16:59.021 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.021 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.021 DEBUG:boto:Params: {} 2024-03-04T16:16:59.021 DEBUG:boto:Token: None 2024-03-04T16:16:59.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.021 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba83ceb95cfc4d5-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.024 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.024 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:59.024 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:59.025 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.025 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:16:59.025 DEBUG:boto:Data: 2024-03-04T16:16:59.025 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.025 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.025 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.025 DEBUG:boto:Params: {} 2024-03-04T16:16:59.025 DEBUG:boto:Token: None 2024-03-04T16:16:59.025 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/. 2024-03-04T16:16:59.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nmGbICFQtnfcpnIHvMF7te2lNzQ= 2024-03-04T16:16:59.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nmGbICFQtnfcpnIHvMF7te2lNzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089d91f8473e642bd-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.027 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:59.028 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:59.028 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.028 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:16:59.028 DEBUG:boto:Data: 2024-03-04T16:16:59.028 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.028 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.028 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.028 DEBUG:boto:Params: {} 2024-03-04T16:16:59.030 DEBUG:boto:Token: None 2024-03-04T16:16:59.031 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/. 2024-03-04T16:16:59.031 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nmGbICFQtnfcpnIHvMF7te2lNzQ= 2024-03-04T16:16:59.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nmGbICFQtnfcpnIHvMF7te2lNzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bbe25a2923a0ab2-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:16:59.034 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:59.034 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:59.034 DEBUG:boto:Method: GET 2024-03-04T16:16:59.034 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:16:59.035 DEBUG:boto:Data: 2024-03-04T16:16:59.035 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.035 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.035 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.035 DEBUG:boto:Params: {} 2024-03-04T16:16:59.036 DEBUG:boto:Token: None 2024-03-04T16:16:59.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/. 2024-03-04T16:16:59.036 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1kELPNciOordXGRtJjc4l0+LMtM= 2024-03-04T16:16:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1kELPNciOordXGRtJjc4l0+LMtM=', 'Content-Length': '0'} 2024-03-04T16:16:59.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000193450230e216e9e-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.080 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:16:59.080 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:16:59.081 DEBUG:boto:Method: GET 2024-03-04T16:16:59.081 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:16:59.081 DEBUG:boto:Data: 2024-03-04T16:16:59.081 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.081 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.081 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.081 DEBUG:boto:Params: {} 2024-03-04T16:16:59.081 DEBUG:boto:Token: None 2024-03-04T16:16:59.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/. 2024-03-04T16:16:59.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1kELPNciOordXGRtJjc4l0+LMtM= 2024-03-04T16:16:59.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1kELPNciOordXGRtJjc4l0+LMtM=', 'Content-Length': '0'} 2024-03-04T16:16:59.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cf4c0f7dd80781a-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.084 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.084 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.084 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.084 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.084 DEBUG:boto:Method: GET 2024-03-04T16:16:59.084 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:16:59.084 DEBUG:boto:Data: 2024-03-04T16:16:59.084 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.084 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.084 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.084 DEBUG:boto:Params: {} 2024-03-04T16:16:59.085 DEBUG:boto:Token: None 2024-03-04T16:16:59.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7ca5ad90cfe0706-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.087 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.088 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.088 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.088 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.088 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:16:59.088 DEBUG:boto:Method: GET 2024-03-04T16:16:59.088 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:16:59.089 DEBUG:boto:Data: 2024-03-04T16:16:59.089 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.089 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.089 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.089 DEBUG:boto:Params: {} 2024-03-04T16:16:59.089 DEBUG:boto:Token: None 2024-03-04T16:16:59.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.089 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2475505e92317bb-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.092 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:59.092 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.092 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.092 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.093 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.093 DEBUG:boto:Method: GET 2024-03-04T16:16:59.093 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.093 DEBUG:boto:Data: 2024-03-04T16:16:59.093 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.093 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.093 DEBUG:boto:Params: {} 2024-03-04T16:16:59.093 DEBUG:boto:Token: None 2024-03-04T16:16:59.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU= 2024-03-04T16:16:59.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU=', 'Content-Length': '0'} 2024-03-04T16:16:59.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e7fbf61afdddfdb-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.095 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.096 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.096 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.096 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.096 DEBUG:boto:Method: GET 2024-03-04T16:16:59.096 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.096 DEBUG:boto:Data: 2024-03-04T16:16:59.096 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.096 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.096 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.096 DEBUG:boto:Params: {} 2024-03-04T16:16:59.097 DEBUG:boto:Token: None 2024-03-04T16:16:59.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.097 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU= 2024-03-04T16:16:59.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU=', 'Content-Length': '0'} 2024-03-04T16:16:59.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f44abbe8dbe7f10-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.099 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.099 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.099 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.099 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.099 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.099 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.099 DEBUG:boto:Data: 2024-03-04T16:16:59.100 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.100 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.100 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.100 DEBUG:boto:Params: {} 2024-03-04T16:16:59.100 DEBUG:boto:Token: None 2024-03-04T16:16:59.100 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.100 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gstAZSJF4j53s3NPvkmSNcrB6ko= 2024-03-04T16:16:59.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gstAZSJF4j53s3NPvkmSNcrB6ko=', 'Content-Length': '0'} 2024-03-04T16:16:59.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db211e524a1a6c9b-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.102 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.102 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.102 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.102 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.103 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.103 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.103 DEBUG:boto:Data: 2024-03-04T16:16:59.103 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.103 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.103 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.103 DEBUG:boto:Params: {} 2024-03-04T16:16:59.103 DEBUG:boto:Token: None 2024-03-04T16:16:59.103 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gstAZSJF4j53s3NPvkmSNcrB6ko= 2024-03-04T16:16:59.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gstAZSJF4j53s3NPvkmSNcrB6ko=', 'Content-Length': '0'} 2024-03-04T16:16:59.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008efde90543a5070-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:59.105 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.106 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.106 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.106 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.106 DEBUG:boto:Method: GET 2024-03-04T16:16:59.106 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.106 DEBUG:boto:Data: 2024-03-04T16:16:59.106 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.106 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.106 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.106 DEBUG:boto:Params: {} 2024-03-04T16:16:59.107 DEBUG:boto:Token: None 2024-03-04T16:16:59.107 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.107 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU= 2024-03-04T16:16:59.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU=', 'Content-Length': '0'} 2024-03-04T16:16:59.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080c39a56d087301f-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.152 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.152 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.152 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.152 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:16:59.152 DEBUG:boto:Method: GET 2024-03-04T16:16:59.152 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.153 DEBUG:boto:Data: 2024-03-04T16:16:59.153 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.153 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.153 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.153 DEBUG:boto:Params: {} 2024-03-04T16:16:59.153 DEBUG:boto:Token: None 2024-03-04T16:16:59.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:16:59.153 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU= 2024-03-04T16:16:59.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o1TQ/US4oNh1KE1oyrhX1VaSKrU=', 'Content-Length': '0'} 2024-03-04T16:16:59.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a61ff2985e629f86-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.155 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.155 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.155 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.156 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.156 DEBUG:boto:Method: GET 2024-03-04T16:16:59.156 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:16:59.156 DEBUG:boto:Data: 2024-03-04T16:16:59.156 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.156 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.156 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.156 DEBUG:boto:Params: {} 2024-03-04T16:16:59.156 DEBUG:boto:Token: None 2024-03-04T16:16:59.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.157 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab6063b15ce362c9-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.159 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.159 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.159 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.160 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.160 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.160 DEBUG:boto:Method: GET 2024-03-04T16:16:59.160 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:16:59.160 DEBUG:boto:Data: 2024-03-04T16:16:59.160 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.160 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.160 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.160 DEBUG:boto:Params: {} 2024-03-04T16:16:59.160 DEBUG:boto:Token: None 2024-03-04T16:16:59.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.161 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087da938e9eb2e995-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.164 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.164 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.164 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.164 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.164 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:16:59.164 DEBUG:boto:Data: 2024-03-04T16:16:59.164 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.165 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.165 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.165 DEBUG:boto:Params: {} 2024-03-04T16:16:59.165 DEBUG:boto:Token: None 2024-03-04T16:16:59.165 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.. 2024-03-04T16:16:59.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TKOPcRkQKdzTjXKDIka16UNSCrk= 2024-03-04T16:16:59.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TKOPcRkQKdzTjXKDIka16UNSCrk=', 'Content-Length': '0'} 2024-03-04T16:16:59.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000502910c42f75841-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.167 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.167 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.167 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.167 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:16:59.167 DEBUG:boto:Data: 2024-03-04T16:16:59.167 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.168 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.168 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.168 DEBUG:boto:Params: {} 2024-03-04T16:16:59.168 DEBUG:boto:Token: None 2024-03-04T16:16:59.168 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.. 2024-03-04T16:16:59.168 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TKOPcRkQKdzTjXKDIka16UNSCrk= 2024-03-04T16:16:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TKOPcRkQKdzTjXKDIka16UNSCrk=', 'Content-Length': '0'} 2024-03-04T16:16:59.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ef6af768f18acc0-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:16:59.170 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.170 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.170 DEBUG:boto:Method: GET 2024-03-04T16:16:59.170 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:16:59.170 DEBUG:boto:Data: 2024-03-04T16:16:59.170 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.170 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.170 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.170 DEBUG:boto:Params: {} 2024-03-04T16:16:59.171 DEBUG:boto:Token: None 2024-03-04T16:16:59.171 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.. 2024-03-04T16:16:59.171 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NgbiSLjUyM53WwGyOVIfAJXjeYI= 2024-03-04T16:16:59.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NgbiSLjUyM53WwGyOVIfAJXjeYI=', 'Content-Length': '0'} 2024-03-04T16:16:59.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba4b65146e8781b4-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.216 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:16:59.216 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:16:59.216 DEBUG:boto:Method: GET 2024-03-04T16:16:59.216 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:16:59.217 DEBUG:boto:Data: 2024-03-04T16:16:59.217 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.217 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.217 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.217 DEBUG:boto:Params: {} 2024-03-04T16:16:59.217 DEBUG:boto:Token: None 2024-03-04T16:16:59.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.. 2024-03-04T16:16:59.217 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NgbiSLjUyM53WwGyOVIfAJXjeYI= 2024-03-04T16:16:59.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NgbiSLjUyM53WwGyOVIfAJXjeYI=', 'Content-Length': '0'} 2024-03-04T16:16:59.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74a185065df01f7-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.219 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.220 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.220 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.220 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.220 DEBUG:boto:Method: GET 2024-03-04T16:16:59.220 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:16:59.220 DEBUG:boto:Data: 2024-03-04T16:16:59.220 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.220 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.220 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.220 DEBUG:boto:Params: {} 2024-03-04T16:16:59.221 DEBUG:boto:Token: None 2024-03-04T16:16:59.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.221 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf581cd20641dab-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.264 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.264 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.265 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.265 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.265 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:16:59.265 DEBUG:boto:Method: GET 2024-03-04T16:16:59.265 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:16:59.265 DEBUG:boto:Data: 2024-03-04T16:16:59.265 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.265 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.265 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.265 DEBUG:boto:Params: {} 2024-03-04T16:16:59.265 DEBUG:boto:Token: None 2024-03-04T16:16:59.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017ec5784db1ab5a2-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.269 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:59.269 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.269 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.269 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.270 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.270 DEBUG:boto:Method: GET 2024-03-04T16:16:59.270 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.270 DEBUG:boto:Data: 2024-03-04T16:16:59.270 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.270 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.270 DEBUG:boto:Params: {} 2024-03-04T16:16:59.270 DEBUG:boto:Token: None 2024-03-04T16:16:59.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ= 2024-03-04T16:16:59.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059344a310b15e742-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.272 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.273 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.273 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.273 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.273 DEBUG:boto:Method: GET 2024-03-04T16:16:59.273 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.273 DEBUG:boto:Data: 2024-03-04T16:16:59.273 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.273 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.273 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.274 DEBUG:boto:Params: {} 2024-03-04T16:16:59.274 DEBUG:boto:Token: None 2024-03-04T16:16:59.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ= 2024-03-04T16:16:59.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0d00f1ef3e3ac74-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.276 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.276 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.276 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.276 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.276 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.277 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.277 DEBUG:boto:Data: 2024-03-04T16:16:59.277 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.277 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.277 DEBUG:boto:Params: {} 2024-03-04T16:16:59.277 DEBUG:boto:Token: None 2024-03-04T16:16:59.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3cgC1wZQ+L4QN3PfFrbiIg2rmPM= 2024-03-04T16:16:59.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3cgC1wZQ+L4QN3PfFrbiIg2rmPM=', 'Content-Length': '0'} 2024-03-04T16:16:59.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041909dce748f5164-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.279 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.279 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.279 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.279 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.280 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.280 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.280 DEBUG:boto:Data: 2024-03-04T16:16:59.280 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.280 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.280 DEBUG:boto:Params: {} 2024-03-04T16:16:59.280 DEBUG:boto:Token: None 2024-03-04T16:16:59.280 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3cgC1wZQ+L4QN3PfFrbiIg2rmPM= 2024-03-04T16:16:59.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3cgC1wZQ+L4QN3PfFrbiIg2rmPM=', 'Content-Length': '0'} 2024-03-04T16:16:59.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a3dcad554a955c9-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:59.282 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.282 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.283 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.283 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.283 DEBUG:boto:Method: GET 2024-03-04T16:16:59.283 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.283 DEBUG:boto:Data: 2024-03-04T16:16:59.283 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.283 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.283 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.283 DEBUG:boto:Params: {} 2024-03-04T16:16:59.283 DEBUG:boto:Token: None 2024-03-04T16:16:59.284 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ= 2024-03-04T16:16:59.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094200b83b72a43a3-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.328 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.328 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.328 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.328 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:16:59.328 DEBUG:boto:Method: GET 2024-03-04T16:16:59.329 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.329 DEBUG:boto:Data: 2024-03-04T16:16:59.329 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.329 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.329 DEBUG:boto:Params: {} 2024-03-04T16:16:59.329 DEBUG:boto:Token: None 2024-03-04T16:16:59.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:16:59.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ= 2024-03-04T16:16:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EGzM/KBMvaus5UulPbkWGKf1LeQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000300592953b9f529f-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.331 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.331 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.332 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.332 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.332 DEBUG:boto:Method: GET 2024-03-04T16:16:59.332 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:16:59.332 DEBUG:boto:Data: 2024-03-04T16:16:59.332 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.332 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.332 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.332 DEBUG:boto:Params: {} 2024-03-04T16:16:59.332 DEBUG:boto:Token: None 2024-03-04T16:16:59.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d98a04148214e72-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.380 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.380 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.381 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.381 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.381 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.381 DEBUG:boto:Method: GET 2024-03-04T16:16:59.381 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:16:59.381 DEBUG:boto:Data: 2024-03-04T16:16:59.381 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.381 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.381 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.381 DEBUG:boto:Params: {} 2024-03-04T16:16:59.382 DEBUG:boto:Token: None 2024-03-04T16:16:59.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d3bac0e5c5097e-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.384 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.385 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.385 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.385 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.385 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:16:59.385 DEBUG:boto:Data: 2024-03-04T16:16:59.385 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.385 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.385 DEBUG:boto:Params: {} 2024-03-04T16:16:59.385 DEBUG:boto:Token: None 2024-03-04T16:16:59.386 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/... 2024-03-04T16:16:59.386 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w8cwlGMuwLVM0pTR8UnV3h6VQoo= 2024-03-04T16:16:59.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w8cwlGMuwLVM0pTR8UnV3h6VQoo=', 'Content-Length': '0'} 2024-03-04T16:16:59.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027352d8b0badf6b1-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.387 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.387 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.388 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.388 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:16:59.388 DEBUG:boto:Data: 2024-03-04T16:16:59.388 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.388 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.388 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.388 DEBUG:boto:Params: {} 2024-03-04T16:16:59.388 DEBUG:boto:Token: None 2024-03-04T16:16:59.388 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/... 2024-03-04T16:16:59.388 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w8cwlGMuwLVM0pTR8UnV3h6VQoo= 2024-03-04T16:16:59.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w8cwlGMuwLVM0pTR8UnV3h6VQoo=', 'Content-Length': '0'} 2024-03-04T16:16:59.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000608480a0b420fb23-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:16:59.391 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.391 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.391 DEBUG:boto:Method: GET 2024-03-04T16:16:59.391 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:16:59.391 DEBUG:boto:Data: 2024-03-04T16:16:59.391 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.391 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.391 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.391 DEBUG:boto:Params: {} 2024-03-04T16:16:59.391 DEBUG:boto:Token: None 2024-03-04T16:16:59.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/... 2024-03-04T16:16:59.392 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7lnKf9Q9QioAKVnxvzXEl5HWcdA= 2024-03-04T16:16:59.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7lnKf9Q9QioAKVnxvzXEl5HWcdA=', 'Content-Length': '0'} 2024-03-04T16:16:59.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003da0c101133c5172-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.436 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:16:59.436 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:16:59.436 DEBUG:boto:Method: GET 2024-03-04T16:16:59.436 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:16:59.436 DEBUG:boto:Data: 2024-03-04T16:16:59.437 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.437 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.437 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.437 DEBUG:boto:Params: {} 2024-03-04T16:16:59.437 DEBUG:boto:Token: None 2024-03-04T16:16:59.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/... 2024-03-04T16:16:59.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7lnKf9Q9QioAKVnxvzXEl5HWcdA= 2024-03-04T16:16:59.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7lnKf9Q9QioAKVnxvzXEl5HWcdA=', 'Content-Length': '0'} 2024-03-04T16:16:59.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b54cfe1ace6d624-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.439 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.439 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.439 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.439 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.439 DEBUG:boto:Method: GET 2024-03-04T16:16:59.439 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:16:59.440 DEBUG:boto:Data: 2024-03-04T16:16:59.440 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.440 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.440 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.440 DEBUG:boto:Params: {} 2024-03-04T16:16:59.440 DEBUG:boto:Token: None 2024-03-04T16:16:59.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aef48de7b75ab90-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.445 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.445 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.445 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.445 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.446 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:16:59.446 DEBUG:boto:Method: GET 2024-03-04T16:16:59.446 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:16:59.446 DEBUG:boto:Data: 2024-03-04T16:16:59.446 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.446 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.446 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.446 DEBUG:boto:Params: {} 2024-03-04T16:16:59.446 DEBUG:boto:Token: None 2024-03-04T16:16:59.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0ad60606aa5184-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.449 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:59.449 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.449 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.450 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.450 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.450 DEBUG:boto:Method: GET 2024-03-04T16:16:59.450 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.450 DEBUG:boto:Data: 2024-03-04T16:16:59.450 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.450 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.450 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.450 DEBUG:boto:Params: {} 2024-03-04T16:16:59.450 DEBUG:boto:Token: None 2024-03-04T16:16:59.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.451 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ= 2024-03-04T16:16:59.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eb6431f8508d658-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.496 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.496 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.496 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.496 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.496 DEBUG:boto:Method: GET 2024-03-04T16:16:59.496 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.497 DEBUG:boto:Data: 2024-03-04T16:16:59.497 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.497 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.497 DEBUG:boto:Params: {} 2024-03-04T16:16:59.497 DEBUG:boto:Token: None 2024-03-04T16:16:59.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ= 2024-03-04T16:16:59.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6fa79d6f0e7a15-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.499 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.499 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.499 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.499 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.500 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.500 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.500 DEBUG:boto:Data: 2024-03-04T16:16:59.500 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.500 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.500 DEBUG:boto:Params: {} 2024-03-04T16:16:59.500 DEBUG:boto:Token: None 2024-03-04T16:16:59.500 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JNrTtcM4qpH0Jz5TAnULg0ljtZo= 2024-03-04T16:16:59.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JNrTtcM4qpH0Jz5TAnULg0ljtZo=', 'Content-Length': '0'} 2024-03-04T16:16:59.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000687cda3717c2ec51-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.503 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.503 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.503 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.503 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.503 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.503 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.503 DEBUG:boto:Data: 2024-03-04T16:16:59.504 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.504 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.504 DEBUG:boto:Params: {} 2024-03-04T16:16:59.504 DEBUG:boto:Token: None 2024-03-04T16:16:59.504 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.504 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JNrTtcM4qpH0Jz5TAnULg0ljtZo= 2024-03-04T16:16:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JNrTtcM4qpH0Jz5TAnULg0ljtZo=', 'Content-Length': '0'} 2024-03-04T16:16:59.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b915dc43b00b42b2-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:59.506 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.506 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.506 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.507 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.507 DEBUG:boto:Method: GET 2024-03-04T16:16:59.507 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.507 DEBUG:boto:Data: 2024-03-04T16:16:59.507 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.507 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.507 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.507 DEBUG:boto:Params: {} 2024-03-04T16:16:59.507 DEBUG:boto:Token: None 2024-03-04T16:16:59.507 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ= 2024-03-04T16:16:59.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001e186f1733e983d-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.556 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.556 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.556 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.556 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.556 DEBUG:boto:Method: GET 2024-03-04T16:16:59.556 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.557 DEBUG:boto:Data: 2024-03-04T16:16:59.557 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.557 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.557 DEBUG:boto:Params: {} 2024-03-04T16:16:59.557 DEBUG:boto:Token: None 2024-03-04T16:16:59.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:16:59.557 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ= 2024-03-04T16:16:59.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gS2nc04lxgtjz40HAGt9Q0ipKzQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000932df4410f48404c-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.559 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.559 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.560 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.560 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.560 DEBUG:boto:Method: GET 2024-03-04T16:16:59.560 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:16:59.560 DEBUG:boto:Data: 2024-03-04T16:16:59.560 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.561 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.561 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.561 DEBUG:boto:Params: {} 2024-03-04T16:16:59.561 DEBUG:boto:Token: None 2024-03-04T16:16:59.561 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.561 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c68067cb16abff-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.564 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.564 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.564 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.565 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.565 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.565 DEBUG:boto:Method: GET 2024-03-04T16:16:59.565 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:16:59.565 DEBUG:boto:Data: 2024-03-04T16:16:59.565 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.565 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.565 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.565 DEBUG:boto:Params: {} 2024-03-04T16:16:59.565 DEBUG:boto:Token: None 2024-03-04T16:16:59.565 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000825706803580011-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.568 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.569 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.569 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.569 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.569 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:16:59.569 DEBUG:boto:Data: 2024-03-04T16:16:59.569 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.569 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.569 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.569 DEBUG:boto:Params: {} 2024-03-04T16:16:59.569 DEBUG:boto:Token: None 2024-03-04T16:16:59.570 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o 2024-03-04T16:16:59.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f6jrc3GLgkzYYTdSLnk3JECIPEY= 2024-03-04T16:16:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f6jrc3GLgkzYYTdSLnk3JECIPEY=', 'Content-Length': '0'} 2024-03-04T16:16:59.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c765e88cc98d40a-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.571 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.571 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.572 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.572 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:16:59.572 DEBUG:boto:Data: 2024-03-04T16:16:59.572 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.572 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.572 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.572 DEBUG:boto:Params: {} 2024-03-04T16:16:59.572 DEBUG:boto:Token: None 2024-03-04T16:16:59.572 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o 2024-03-04T16:16:59.572 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f6jrc3GLgkzYYTdSLnk3JECIPEY= 2024-03-04T16:16:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f6jrc3GLgkzYYTdSLnk3JECIPEY=', 'Content-Length': '0'} 2024-03-04T16:16:59.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec3abf6dd8f4643e-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:16:59.574 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.574 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.574 DEBUG:boto:Method: GET 2024-03-04T16:16:59.575 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:16:59.575 DEBUG:boto:Data: 2024-03-04T16:16:59.575 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.575 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.575 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.575 DEBUG:boto:Params: {} 2024-03-04T16:16:59.575 DEBUG:boto:Token: None 2024-03-04T16:16:59.575 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o 2024-03-04T16:16:59.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q200N0A3Y/AkoB4vnJpoehOB5Ao= 2024-03-04T16:16:59.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q200N0A3Y/AkoB4vnJpoehOB5Ao=', 'Content-Length': '0'} 2024-03-04T16:16:59.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e2fc7b0ccfffed6-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.624 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:16:59.624 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:16:59.624 DEBUG:boto:Method: GET 2024-03-04T16:16:59.624 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:16:59.624 DEBUG:boto:Data: 2024-03-04T16:16:59.625 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.625 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.625 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.625 DEBUG:boto:Params: {} 2024-03-04T16:16:59.625 DEBUG:boto:Token: None 2024-03-04T16:16:59.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o 2024-03-04T16:16:59.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q200N0A3Y/AkoB4vnJpoehOB5Ao= 2024-03-04T16:16:59.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q200N0A3Y/AkoB4vnJpoehOB5Ao=', 'Content-Length': '0'} 2024-03-04T16:16:59.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7730c8d86b16679-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.627 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.627 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.627 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.628 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.628 DEBUG:boto:Method: GET 2024-03-04T16:16:59.628 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:16:59.628 DEBUG:boto:Data: 2024-03-04T16:16:59.628 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.628 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.628 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.628 DEBUG:boto:Params: {} 2024-03-04T16:16:59.628 DEBUG:boto:Token: None 2024-03-04T16:16:59.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.629 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5af3ddf115cb04-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.672 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.672 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.672 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.673 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.673 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:16:59.673 DEBUG:boto:Method: GET 2024-03-04T16:16:59.673 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:16:59.673 DEBUG:boto:Data: 2024-03-04T16:16:59.673 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.673 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.673 DEBUG:boto:Params: {} 2024-03-04T16:16:59.673 DEBUG:boto:Token: None 2024-03-04T16:16:59.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab191d954f83b77-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.676 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:59.677 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.677 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.677 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.677 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.677 DEBUG:boto:Method: GET 2024-03-04T16:16:59.677 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.677 DEBUG:boto:Data: 2024-03-04T16:16:59.677 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.677 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.677 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.677 DEBUG:boto:Params: {} 2024-03-04T16:16:59.677 DEBUG:boto:Token: None 2024-03-04T16:16:59.678 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw= 2024-03-04T16:16:59.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw=', 'Content-Length': '0'} 2024-03-04T16:16:59.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051ce53ffbaa1a282-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.679 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.679 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.680 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.680 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.680 DEBUG:boto:Method: GET 2024-03-04T16:16:59.680 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.680 DEBUG:boto:Data: 2024-03-04T16:16:59.680 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.680 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.680 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.680 DEBUG:boto:Params: {} 2024-03-04T16:16:59.680 DEBUG:boto:Token: None 2024-03-04T16:16:59.681 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.681 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw= 2024-03-04T16:16:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw=', 'Content-Length': '0'} 2024-03-04T16:16:59.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b32a158f53356ef2-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.683 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.683 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.683 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.683 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.683 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.683 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.683 DEBUG:boto:Data: 2024-03-04T16:16:59.683 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.683 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.683 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.683 DEBUG:boto:Params: {} 2024-03-04T16:16:59.683 DEBUG:boto:Token: None 2024-03-04T16:16:59.684 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.684 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wQQCNCHbyWc+DcovmD1D3eGWrqw= 2024-03-04T16:16:59.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wQQCNCHbyWc+DcovmD1D3eGWrqw=', 'Content-Length': '0'} 2024-03-04T16:16:59.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e15e2bcd55a6498f-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.686 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.686 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.686 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.686 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.686 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.686 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.686 DEBUG:boto:Data: 2024-03-04T16:16:59.686 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.686 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.686 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.687 DEBUG:boto:Params: {} 2024-03-04T16:16:59.687 DEBUG:boto:Token: None 2024-03-04T16:16:59.687 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.687 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wQQCNCHbyWc+DcovmD1D3eGWrqw= 2024-03-04T16:16:59.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wQQCNCHbyWc+DcovmD1D3eGWrqw=', 'Content-Length': '0'} 2024-03-04T16:16:59.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8d6c6acd5c3c663-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:59.689 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.689 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.689 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.689 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.689 DEBUG:boto:Method: GET 2024-03-04T16:16:59.689 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.689 DEBUG:boto:Data: 2024-03-04T16:16:59.689 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.690 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.690 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.690 DEBUG:boto:Params: {} 2024-03-04T16:16:59.690 DEBUG:boto:Token: None 2024-03-04T16:16:59.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw= 2024-03-04T16:16:59.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw=', 'Content-Length': '0'} 2024-03-04T16:16:59.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b443717a9e934300-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.736 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.736 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.736 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.736 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.736 DEBUG:boto:Method: GET 2024-03-04T16:16:59.737 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.737 DEBUG:boto:Data: 2024-03-04T16:16:59.737 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.737 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.737 DEBUG:boto:Params: {} 2024-03-04T16:16:59.737 DEBUG:boto:Token: None 2024-03-04T16:16:59.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:16:59.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw= 2024-03-04T16:16:59.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M9o7AaoPC7mqptGoz5zetb1/FMw=', 'Content-Length': '0'} 2024-03-04T16:16:59.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a3b92e391644369-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.739 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.739 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.739 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.739 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.739 DEBUG:boto:Method: GET 2024-03-04T16:16:59.739 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.740 DEBUG:boto:Data: 2024-03-04T16:16:59.740 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.740 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.740 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.740 DEBUG:boto:Params: {} 2024-03-04T16:16:59.740 DEBUG:boto:Token: None 2024-03-04T16:16:59.740 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.740 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368677f4124143c1-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.743 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.743 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.743 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.743 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.743 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.744 DEBUG:boto:Method: GET 2024-03-04T16:16:59.744 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.744 DEBUG:boto:Data: 2024-03-04T16:16:59.744 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.744 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.744 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.744 DEBUG:boto:Params: {} 2024-03-04T16:16:59.744 DEBUG:boto:Token: None 2024-03-04T16:16:59.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.745 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005262796bb49a76ac-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.747 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.747 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.747 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.747 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.747 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:16:59.748 DEBUG:boto:Data: 2024-03-04T16:16:59.748 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.748 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.748 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.748 DEBUG:boto:Params: {} 2024-03-04T16:16:59.748 DEBUG:boto:Token: None 2024-03-04T16:16:59.748 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o. 2024-03-04T16:16:59.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4a+ye7QHFtBYI2wvyF+KS04RTUU= 2024-03-04T16:16:59.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4a+ye7QHFtBYI2wvyF+KS04RTUU=', 'Content-Length': '0'} 2024-03-04T16:16:59.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c876d716c2912ff-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.750 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.750 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.750 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.750 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:16:59.750 DEBUG:boto:Data: 2024-03-04T16:16:59.750 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.750 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.751 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.751 DEBUG:boto:Params: {} 2024-03-04T16:16:59.751 DEBUG:boto:Token: None 2024-03-04T16:16:59.751 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o. 2024-03-04T16:16:59.751 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4a+ye7QHFtBYI2wvyF+KS04RTUU= 2024-03-04T16:16:59.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4a+ye7QHFtBYI2wvyF+KS04RTUU=', 'Content-Length': '0'} 2024-03-04T16:16:59.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074661010dfbb178d-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:16:59.753 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.753 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.753 DEBUG:boto:Method: GET 2024-03-04T16:16:59.753 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:16:59.753 DEBUG:boto:Data: 2024-03-04T16:16:59.754 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.754 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.754 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.754 DEBUG:boto:Params: {} 2024-03-04T16:16:59.754 DEBUG:boto:Token: None 2024-03-04T16:16:59.754 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o. 2024-03-04T16:16:59.754 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R20dhL6RAdzo52fvXJKWAemkpR4= 2024-03-04T16:16:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R20dhL6RAdzo52fvXJKWAemkpR4=', 'Content-Length': '0'} 2024-03-04T16:16:59.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009374e457f0a56c3b-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.796 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:16:59.796 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:16:59.796 DEBUG:boto:Method: GET 2024-03-04T16:16:59.796 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:16:59.796 DEBUG:boto:Data: 2024-03-04T16:16:59.797 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.797 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.797 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.797 DEBUG:boto:Params: {} 2024-03-04T16:16:59.797 DEBUG:boto:Token: None 2024-03-04T16:16:59.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/.o. 2024-03-04T16:16:59.797 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R20dhL6RAdzo52fvXJKWAemkpR4= 2024-03-04T16:16:59.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R20dhL6RAdzo52fvXJKWAemkpR4=', 'Content-Length': '0'} 2024-03-04T16:16:59.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005da1bf1155889632-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.799 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.799 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.799 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.800 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.800 DEBUG:boto:Method: GET 2024-03-04T16:16:59.800 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.800 DEBUG:boto:Data: 2024-03-04T16:16:59.800 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.800 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.800 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.800 DEBUG:boto:Params: {} 2024-03-04T16:16:59.800 DEBUG:boto:Token: None 2024-03-04T16:16:59.800 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.801 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5cb0cf35455d515-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.844 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.844 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.844 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.844 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.845 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.845 DEBUG:boto:Method: GET 2024-03-04T16:16:59.845 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:16:59.845 DEBUG:boto:Data: 2024-03-04T16:16:59.845 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.845 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.845 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.845 DEBUG:boto:Params: {} 2024-03-04T16:16:59.845 DEBUG:boto:Token: None 2024-03-04T16:16:59.845 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1290b3696c078e-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.848 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:59.848 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.848 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.848 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.848 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.848 DEBUG:boto:Method: GET 2024-03-04T16:16:59.848 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.849 DEBUG:boto:Data: 2024-03-04T16:16:59.849 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.849 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.849 DEBUG:boto:Params: {} 2024-03-04T16:16:59.849 DEBUG:boto:Token: None 2024-03-04T16:16:59.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY= 2024-03-04T16:16:59.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY=', 'Content-Length': '0'} 2024-03-04T16:16:59.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1b9481ccf78c4ef-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.892 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.892 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.892 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.892 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.892 DEBUG:boto:Method: GET 2024-03-04T16:16:59.893 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.893 DEBUG:boto:Data: 2024-03-04T16:16:59.893 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.893 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.893 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.893 DEBUG:boto:Params: {} 2024-03-04T16:16:59.893 DEBUG:boto:Token: None 2024-03-04T16:16:59.893 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.893 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY= 2024-03-04T16:16:59.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY=', 'Content-Length': '0'} 2024-03-04T16:16:59.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b364fbae87fdd124-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.895 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.895 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.895 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.896 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.896 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.896 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.896 DEBUG:boto:Data: 2024-03-04T16:16:59.896 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.896 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.896 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.896 DEBUG:boto:Params: {} 2024-03-04T16:16:59.896 DEBUG:boto:Token: None 2024-03-04T16:16:59.896 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.897 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vc7pmq9rumwmVO7r3N+MQBdKwAQ= 2024-03-04T16:16:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vc7pmq9rumwmVO7r3N+MQBdKwAQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d482528b9deca94a-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.898 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.898 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.898 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.899 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.899 DEBUG:boto:Method: HEAD 2024-03-04T16:16:59.899 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.899 DEBUG:boto:Data: 2024-03-04T16:16:59.899 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.899 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.899 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.899 DEBUG:boto:Params: {} 2024-03-04T16:16:59.899 DEBUG:boto:Token: None 2024-03-04T16:16:59.899 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.900 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vc7pmq9rumwmVO7r3N+MQBdKwAQ= 2024-03-04T16:16:59.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vc7pmq9rumwmVO7r3N+MQBdKwAQ=', 'Content-Length': '0'} 2024-03-04T16:16:59.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb20228508f7dd2b-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:16:59.901 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.901 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.902 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.902 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.902 DEBUG:boto:Method: GET 2024-03-04T16:16:59.902 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.902 DEBUG:boto:Data: 2024-03-04T16:16:59.902 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.902 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.902 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.902 DEBUG:boto:Params: {} 2024-03-04T16:16:59.902 DEBUG:boto:Token: None 2024-03-04T16:16:59.902 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.903 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY= 2024-03-04T16:16:59.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY=', 'Content-Length': '0'} 2024-03-04T16:16:59.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001931b8e339fc37c1-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.948 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:16:59.948 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:16:59.948 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.948 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.948 DEBUG:boto:Method: GET 2024-03-04T16:16:59.949 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.949 DEBUG:boto:Data: 2024-03-04T16:16:59.949 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.949 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.949 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.949 DEBUG:boto:Params: {} 2024-03-04T16:16:59.949 DEBUG:boto:Token: None 2024-03-04T16:16:59.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:16:59.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY= 2024-03-04T16:16:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V9hdjVlXpTZkUQmgv4yUuHrKJyY=', 'Content-Length': '0'} 2024-03-04T16:16:59.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029f46a9fb3371eef-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.951 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.952 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.952 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.952 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.952 DEBUG:boto:Method: GET 2024-03-04T16:16:59.952 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.952 DEBUG:boto:Data: 2024-03-04T16:16:59.952 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.952 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:16:59.952 DEBUG:boto:Port: 8000 2024-03-04T16:16:59.952 DEBUG:boto:Params: {} 2024-03-04T16:16:59.953 DEBUG:boto:Token: None 2024-03-04T16:16:59.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.953 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:16:59.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765b181a53ac48ce-0065e5f3fb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:16:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:16:59.996 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:16:59.997 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:16:59.997 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:16:59.997 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.997 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.997 DEBUG:boto:Method: GET 2024-03-04T16:16:59.997 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:16:59.997 DEBUG:boto:Data: 2024-03-04T16:16:59.997 DEBUG:boto:Headers: {} 2024-03-04T16:16:59.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:16:59.997 DEBUG:boto:Port: 8001 2024-03-04T16:16:59.997 DEBUG:boto:Params: {} 2024-03-04T16:16:59.998 DEBUG:boto:Token: None 2024-03-04T16:16:59.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:16:59 GMT /sbnvju-12/ 2024-03-04T16:16:59.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y= 2024-03-04T16:16:59.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:16:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Whp80u8Uu2KhJo7utD0vwYejh/Y=', 'Content-Length': '0'} 2024-03-04T16:17:00.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c3d7f6e421905c-0065e5f3fb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.000 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.001 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:00.001 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:00.001 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.001 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:00.001 DEBUG:boto:Data: 2024-03-04T16:17:00.001 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.001 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.001 DEBUG:boto:Params: {} 2024-03-04T16:17:00.001 DEBUG:boto:Token: None 2024-03-04T16:17:00.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%3A 2024-03-04T16:17:00.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xALBePUtanXICgTvK7gO8y4B4Zk= 2024-03-04T16:17:00.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xALBePUtanXICgTvK7gO8y4B4Zk=', 'Content-Length': '0'} 2024-03-04T16:17:00.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7b7648a53796db4-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.003 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:00.004 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:00.004 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.004 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:00.004 DEBUG:boto:Data: 2024-03-04T16:17:00.004 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.004 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.004 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.004 DEBUG:boto:Params: {} 2024-03-04T16:17:00.004 DEBUG:boto:Token: None 2024-03-04T16:17:00.005 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%3A 2024-03-04T16:17:00.005 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xALBePUtanXICgTvK7gO8y4B4Zk= 2024-03-04T16:17:00.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xALBePUtanXICgTvK7gO8y4B4Zk=', 'Content-Length': '0'} 2024-03-04T16:17:00.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0712e770b81a948-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:17:00.007 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:00.007 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:00.007 DEBUG:boto:Method: GET 2024-03-04T16:17:00.007 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:00.007 DEBUG:boto:Data: 2024-03-04T16:17:00.007 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.007 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.007 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.007 DEBUG:boto:Params: {} 2024-03-04T16:17:00.007 DEBUG:boto:Token: None 2024-03-04T16:17:00.007 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%3A 2024-03-04T16:17:00.008 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWasHJH9K1Qu5GfScOiP0NWnt0w= 2024-03-04T16:17:00.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWasHJH9K1Qu5GfScOiP0NWnt0w=', 'Content-Length': '0'} 2024-03-04T16:17:00.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001903996512896180-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.052 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:00.052 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:00.052 DEBUG:boto:Method: GET 2024-03-04T16:17:00.052 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:00.052 DEBUG:boto:Data: 2024-03-04T16:17:00.053 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.053 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.053 DEBUG:boto:Params: {} 2024-03-04T16:17:00.053 DEBUG:boto:Token: None 2024-03-04T16:17:00.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%3A 2024-03-04T16:17:00.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWasHJH9K1Qu5GfScOiP0NWnt0w= 2024-03-04T16:17:00.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWasHJH9K1Qu5GfScOiP0NWnt0w=', 'Content-Length': '0'} 2024-03-04T16:17:00.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad758d3e4b448eae-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.055 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.055 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.055 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.055 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.055 DEBUG:boto:Method: GET 2024-03-04T16:17:00.055 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.055 DEBUG:boto:Data: 2024-03-04T16:17:00.056 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.056 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.056 DEBUG:boto:Params: {} 2024-03-04T16:17:00.056 DEBUG:boto:Token: None 2024-03-04T16:17:00.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.056 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9cf02f8445f232d-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.058 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.059 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.059 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.059 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.059 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.059 DEBUG:boto:Method: GET 2024-03-04T16:17:00.059 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:00.059 DEBUG:boto:Data: 2024-03-04T16:17:00.059 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.059 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.059 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.059 DEBUG:boto:Params: {} 2024-03-04T16:17:00.060 DEBUG:boto:Token: None 2024-03-04T16:17:00.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.060 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c626a5610df6c75-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.062 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:00.062 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.063 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.063 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.063 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.063 DEBUG:boto:Method: GET 2024-03-04T16:17:00.063 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.063 DEBUG:boto:Data: 2024-03-04T16:17:00.063 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.063 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.063 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.063 DEBUG:boto:Params: {} 2024-03-04T16:17:00.063 DEBUG:boto:Token: None 2024-03-04T16:17:00.063 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.064 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI= 2024-03-04T16:17:00.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI=', 'Content-Length': '0'} 2024-03-04T16:17:00.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bdc334d5be82d30-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.108 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.108 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.108 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.108 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.109 DEBUG:boto:Method: GET 2024-03-04T16:17:00.109 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.109 DEBUG:boto:Data: 2024-03-04T16:17:00.109 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.109 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.109 DEBUG:boto:Params: {} 2024-03-04T16:17:00.109 DEBUG:boto:Token: None 2024-03-04T16:17:00.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI= 2024-03-04T16:17:00.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI=', 'Content-Length': '0'} 2024-03-04T16:17:00.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080f75670d8bcddd8-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.112 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.112 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.112 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.112 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.112 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.112 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.112 DEBUG:boto:Data: 2024-03-04T16:17:00.113 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.113 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.113 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.113 DEBUG:boto:Params: {} 2024-03-04T16:17:00.113 DEBUG:boto:Token: None 2024-03-04T16:17:00.113 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.113 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G57PXL2RcI+0s0mT9U8bxMcTipY= 2024-03-04T16:17:00.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G57PXL2RcI+0s0mT9U8bxMcTipY=', 'Content-Length': '0'} 2024-03-04T16:17:00.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8e4260c716ed413-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.115 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.115 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.115 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.115 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.115 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.115 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.115 DEBUG:boto:Data: 2024-03-04T16:17:00.115 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.116 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.116 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.116 DEBUG:boto:Params: {} 2024-03-04T16:17:00.116 DEBUG:boto:Token: None 2024-03-04T16:17:00.116 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.116 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G57PXL2RcI+0s0mT9U8bxMcTipY= 2024-03-04T16:17:00.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G57PXL2RcI+0s0mT9U8bxMcTipY=', 'Content-Length': '0'} 2024-03-04T16:17:00.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e500dcffa35a5d63-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:00.118 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.118 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.118 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.118 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.118 DEBUG:boto:Method: GET 2024-03-04T16:17:00.118 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.118 DEBUG:boto:Data: 2024-03-04T16:17:00.118 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.119 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.119 DEBUG:boto:Params: {} 2024-03-04T16:17:00.119 DEBUG:boto:Token: None 2024-03-04T16:17:00.119 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.119 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI= 2024-03-04T16:17:00.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI=', 'Content-Length': '0'} 2024-03-04T16:17:00.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000736706ad1434339f-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.164 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.164 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.164 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.165 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.165 DEBUG:boto:Method: GET 2024-03-04T16:17:00.165 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.165 DEBUG:boto:Data: 2024-03-04T16:17:00.165 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.165 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.165 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.165 DEBUG:boto:Params: {} 2024-03-04T16:17:00.165 DEBUG:boto:Token: None 2024-03-04T16:17:00.165 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:00.166 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI= 2024-03-04T16:17:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xrEOGUFAaaSdW5ajIbmuA/By3eI=', 'Content-Length': '0'} 2024-03-04T16:17:00.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab00a25b41463be5-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.168 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.168 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.168 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.168 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.168 DEBUG:boto:Method: GET 2024-03-04T16:17:00.168 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.168 DEBUG:boto:Data: 2024-03-04T16:17:00.168 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.168 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.169 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.169 DEBUG:boto:Params: {} 2024-03-04T16:17:00.169 DEBUG:boto:Token: None 2024-03-04T16:17:00.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1f966f891de27c-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.172 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.172 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.172 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.172 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.172 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.172 DEBUG:boto:Method: GET 2024-03-04T16:17:00.172 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.173 DEBUG:boto:Data: 2024-03-04T16:17:00.173 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.173 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.173 DEBUG:boto:Params: {} 2024-03-04T16:17:00.173 DEBUG:boto:Token: None 2024-03-04T16:17:00.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092967bca60dac9f0-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.176 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.176 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.176 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.176 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.176 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:00.176 DEBUG:boto:Data: 2024-03-04T16:17:00.176 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.176 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.177 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.177 DEBUG:boto:Params: {} 2024-03-04T16:17:00.177 DEBUG:boto:Token: None 2024-03-04T16:17:00.177 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj 2024-03-04T16:17:00.177 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/ISd5Acf7E4mQ5qRW4C0yN+RHkQ= 2024-03-04T16:17:00.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/ISd5Acf7E4mQ5qRW4C0yN+RHkQ=', 'Content-Length': '0'} 2024-03-04T16:17:00.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaa8d2f738c41ca3-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.179 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.179 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.179 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.179 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:00.179 DEBUG:boto:Data: 2024-03-04T16:17:00.179 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.179 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.179 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.179 DEBUG:boto:Params: {} 2024-03-04T16:17:00.179 DEBUG:boto:Token: None 2024-03-04T16:17:00.180 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj 2024-03-04T16:17:00.180 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/ISd5Acf7E4mQ5qRW4C0yN+RHkQ= 2024-03-04T16:17:00.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/ISd5Acf7E4mQ5qRW4C0yN+RHkQ=', 'Content-Length': '0'} 2024-03-04T16:17:00.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e80b5e36e48c1ab-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:00.182 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.182 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.182 DEBUG:boto:Method: GET 2024-03-04T16:17:00.182 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:00.182 DEBUG:boto:Data: 2024-03-04T16:17:00.182 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.182 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.182 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.182 DEBUG:boto:Params: {} 2024-03-04T16:17:00.182 DEBUG:boto:Token: None 2024-03-04T16:17:00.183 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj 2024-03-04T16:17:00.183 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xYLi74dYJmw8tERnNoKcOP2lj3Y= 2024-03-04T16:17:00.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xYLi74dYJmw8tERnNoKcOP2lj3Y=', 'Content-Length': '0'} 2024-03-04T16:17:00.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db51f424d5a2f326-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.228 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:00.228 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:00.229 DEBUG:boto:Method: GET 2024-03-04T16:17:00.229 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:00.229 DEBUG:boto:Data: 2024-03-04T16:17:00.229 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.229 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.229 DEBUG:boto:Params: {} 2024-03-04T16:17:00.229 DEBUG:boto:Token: None 2024-03-04T16:17:00.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/_myobj 2024-03-04T16:17:00.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xYLi74dYJmw8tERnNoKcOP2lj3Y= 2024-03-04T16:17:00.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xYLi74dYJmw8tERnNoKcOP2lj3Y=', 'Content-Length': '0'} 2024-03-04T16:17:00.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a10cdbec225a99a-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.232 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.232 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.232 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.232 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.232 DEBUG:boto:Method: GET 2024-03-04T16:17:00.232 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.232 DEBUG:boto:Data: 2024-03-04T16:17:00.233 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.233 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.233 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.233 DEBUG:boto:Params: {} 2024-03-04T16:17:00.233 DEBUG:boto:Token: None 2024-03-04T16:17:00.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8933d4232eb873d-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.235 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.236 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.236 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.236 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.236 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.236 DEBUG:boto:Method: GET 2024-03-04T16:17:00.236 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:00.236 DEBUG:boto:Data: 2024-03-04T16:17:00.236 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.236 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.237 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.237 DEBUG:boto:Params: {} 2024-03-04T16:17:00.237 DEBUG:boto:Token: None 2024-03-04T16:17:00.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f7ccde097a9d87-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.239 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:00.240 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.240 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.240 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.240 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.240 DEBUG:boto:Method: GET 2024-03-04T16:17:00.240 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.240 DEBUG:boto:Data: 2024-03-04T16:17:00.240 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.240 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.240 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.240 DEBUG:boto:Params: {} 2024-03-04T16:17:00.241 DEBUG:boto:Token: None 2024-03-04T16:17:00.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw= 2024-03-04T16:17:00.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw=', 'Content-Length': '0'} 2024-03-04T16:17:00.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f9baf026d9098f2-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.243 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.243 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.243 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.243 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.243 DEBUG:boto:Method: GET 2024-03-04T16:17:00.243 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.243 DEBUG:boto:Data: 2024-03-04T16:17:00.243 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.243 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.243 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.243 DEBUG:boto:Params: {} 2024-03-04T16:17:00.244 DEBUG:boto:Token: None 2024-03-04T16:17:00.244 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.244 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw= 2024-03-04T16:17:00.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw=', 'Content-Length': '0'} 2024-03-04T16:17:00.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063f3baa41cf78e2c-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.246 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.246 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.246 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.246 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.246 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.246 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.247 DEBUG:boto:Data: 2024-03-04T16:17:00.247 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.247 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.247 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.247 DEBUG:boto:Params: {} 2024-03-04T16:17:00.247 DEBUG:boto:Token: None 2024-03-04T16:17:00.247 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.247 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kQaFKxwieBvAJJIvu5S9XtHJfRg= 2024-03-04T16:17:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kQaFKxwieBvAJJIvu5S9XtHJfRg=', 'Content-Length': '0'} 2024-03-04T16:17:00.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000385d91482c682b0-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.249 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.249 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.249 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.249 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.249 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.249 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.249 DEBUG:boto:Data: 2024-03-04T16:17:00.249 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.250 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.250 DEBUG:boto:Params: {} 2024-03-04T16:17:00.250 DEBUG:boto:Token: None 2024-03-04T16:17:00.250 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.250 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kQaFKxwieBvAJJIvu5S9XtHJfRg= 2024-03-04T16:17:00.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kQaFKxwieBvAJJIvu5S9XtHJfRg=', 'Content-Length': '0'} 2024-03-04T16:17:00.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c360318b332bcc12-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:00.252 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.252 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.252 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.252 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.252 DEBUG:boto:Method: GET 2024-03-04T16:17:00.252 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.252 DEBUG:boto:Data: 2024-03-04T16:17:00.253 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.253 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.253 DEBUG:boto:Params: {} 2024-03-04T16:17:00.253 DEBUG:boto:Token: None 2024-03-04T16:17:00.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.253 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw= 2024-03-04T16:17:00.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw=', 'Content-Length': '0'} 2024-03-04T16:17:00.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bbec4a203c5111e-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.296 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.296 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.296 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.296 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.296 DEBUG:boto:Method: GET 2024-03-04T16:17:00.297 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.297 DEBUG:boto:Data: 2024-03-04T16:17:00.297 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.297 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.297 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.297 DEBUG:boto:Params: {} 2024-03-04T16:17:00.297 DEBUG:boto:Token: None 2024-03-04T16:17:00.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:00.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw= 2024-03-04T16:17:00.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1Dd8rWidTBQrLxy5H/8SvkinAw=', 'Content-Length': '0'} 2024-03-04T16:17:00.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e90b0ab803b1025d-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.299 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.299 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.299 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.300 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.300 DEBUG:boto:Method: GET 2024-03-04T16:17:00.300 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.300 DEBUG:boto:Data: 2024-03-04T16:17:00.300 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.300 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.300 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.300 DEBUG:boto:Params: {} 2024-03-04T16:17:00.300 DEBUG:boto:Token: None 2024-03-04T16:17:00.300 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.301 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008091cfa4476940d3-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.303 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.303 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.303 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.303 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.303 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.303 DEBUG:boto:Method: GET 2024-03-04T16:17:00.303 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.304 DEBUG:boto:Data: 2024-03-04T16:17:00.304 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.304 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.304 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.304 DEBUG:boto:Params: {} 2024-03-04T16:17:00.304 DEBUG:boto:Token: None 2024-03-04T16:17:00.304 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.304 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa5cd65cc0c64b8-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.307 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.307 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.307 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.307 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.307 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:00.307 DEBUG:boto:Data: 2024-03-04T16:17:00.307 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.307 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.308 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.308 DEBUG:boto:Params: {} 2024-03-04T16:17:00.308 DEBUG:boto:Token: None 2024-03-04T16:17:00.308 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj 2024-03-04T16:17:00.308 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xxMNSatohBuyXwaodnJj+B/Q9fo= 2024-03-04T16:17:00.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xxMNSatohBuyXwaodnJj+B/Q9fo=', 'Content-Length': '0'} 2024-03-04T16:17:00.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8996afe7e1c8751-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.310 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.310 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.310 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.310 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:00.310 DEBUG:boto:Data: 2024-03-04T16:17:00.310 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.310 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.310 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.310 DEBUG:boto:Params: {} 2024-03-04T16:17:00.310 DEBUG:boto:Token: None 2024-03-04T16:17:00.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj 2024-03-04T16:17:00.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xxMNSatohBuyXwaodnJj+B/Q9fo= 2024-03-04T16:17:00.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xxMNSatohBuyXwaodnJj+B/Q9fo=', 'Content-Length': '0'} 2024-03-04T16:17:00.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063a049a5d206a8c5-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:00.312 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.313 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.313 DEBUG:boto:Method: GET 2024-03-04T16:17:00.313 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:00.313 DEBUG:boto:Data: 2024-03-04T16:17:00.313 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.313 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.313 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.313 DEBUG:boto:Params: {} 2024-03-04T16:17:00.313 DEBUG:boto:Token: None 2024-03-04T16:17:00.313 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj 2024-03-04T16:17:00.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HNaOV0Cr8k7RHuGXk8m1xyfXEkc= 2024-03-04T16:17:00.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HNaOV0Cr8k7RHuGXk8m1xyfXEkc=', 'Content-Length': '0'} 2024-03-04T16:17:00.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe0512f806303a61-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.356 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:00.356 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:00.356 DEBUG:boto:Method: GET 2024-03-04T16:17:00.356 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:00.356 DEBUG:boto:Data: 2024-03-04T16:17:00.356 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.357 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.357 DEBUG:boto:Params: {} 2024-03-04T16:17:00.357 DEBUG:boto:Token: None 2024-03-04T16:17:00.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/myobj 2024-03-04T16:17:00.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HNaOV0Cr8k7RHuGXk8m1xyfXEkc= 2024-03-04T16:17:00.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HNaOV0Cr8k7RHuGXk8m1xyfXEkc=', 'Content-Length': '0'} 2024-03-04T16:17:00.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd26d4d72c26ee11-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.359 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.359 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.359 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.359 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.359 DEBUG:boto:Method: GET 2024-03-04T16:17:00.360 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.360 DEBUG:boto:Data: 2024-03-04T16:17:00.360 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.360 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:00.360 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.360 DEBUG:boto:Params: {} 2024-03-04T16:17:00.360 DEBUG:boto:Token: None 2024-03-04T16:17:00.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.360 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5b073fd108a7eb-0065e5f3fc-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.362 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.363 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.363 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.363 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.363 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.363 DEBUG:boto:Method: GET 2024-03-04T16:17:00.363 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:00.363 DEBUG:boto:Data: 2024-03-04T16:17:00.363 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.364 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.364 DEBUG:boto:Params: {} 2024-03-04T16:17:00.364 DEBUG:boto:Token: None 2024-03-04T16:17:00.364 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.364 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026295a95b14f5a27-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.367 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.367 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-12 zones={test-zone1, test-zone3} 2024-03-04T16:17:00.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:00.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:00.367 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:17:00.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:00.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:00.483+0000 7f6b64133800 1 bucket sync caught up with source: 2024-03-04T16:17:00.484 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.12.6, , 00000000001.15.6, , 00000000001.17.6, 00000000002.17.6, 00000000001.9.6, 00000000001.14.6, 00000000001.13.6, ] 2024-03-04T16:17:00.484 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.12.6, , 00000000001.15.6, , 00000000001.17.6, 00000000002.17.6, 00000000001.9.6, 00000000001.14.6, 00000000001.13.6, ] 2024-03-04T16:17:00.484 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:00.483+0000 7f6b64133800 0 bucket checkpoint complete 2024-03-04T16:17:00.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-12 zones={test-zone2, test-zone3} 2024-03-04T16:17:00.503 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.503 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.503 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.503 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:17:00.503 DEBUG:boto:Data: 2024-03-04T16:17:00.503 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.504 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.504 DEBUG:boto:Params: {} 2024-03-04T16:17:00.504 DEBUG:boto:Token: None 2024-03-04T16:17:00.504 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.504 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LMr/v9rDhmzaDrRbTNMklZv4s0g= 2024-03-04T16:17:00.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LMr/v9rDhmzaDrRbTNMklZv4s0g=', 'Content-Length': '0'} 2024-03-04T16:17:00.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000906fb14ae805c55-0065e5f3fc-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.507 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.507 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.507 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.507 DEBUG:boto:Path: /sbnvju-12/ 2024-03-04T16:17:00.507 DEBUG:boto:Data: 2024-03-04T16:17:00.507 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.507 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.507 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.507 DEBUG:boto:Params: {} 2024-03-04T16:17:00.508 DEBUG:boto:Token: None 2024-03-04T16:17:00.508 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LMr/v9rDhmzaDrRbTNMklZv4s0g= 2024-03-04T16:17:00.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LMr/v9rDhmzaDrRbTNMklZv4s0g=', 'Content-Length': '0'} 2024-03-04T16:17:00.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5b450308b2b6b36-0065e5f3fc-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:17:00.510 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.510 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.510 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.510 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.510 DEBUG:boto:Method: GET 2024-03-04T16:17:00.511 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.511 DEBUG:boto:Data: 2024-03-04T16:17:00.511 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.511 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.511 DEBUG:boto:Params: {} 2024-03-04T16:17:00.511 DEBUG:boto:Token: None 2024-03-04T16:17:00.511 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/?versions 2024-03-04T16:17:00.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk= 2024-03-04T16:17:00.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk=', 'Content-Length': '0'} 2024-03-04T16:17:00.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9df864e5c5dc39-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.514 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:00.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:17:00.516 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.516 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.516 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.516 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.516 DEBUG:boto:Method: GET 2024-03-04T16:17:00.516 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.516 DEBUG:boto:Data: 2024-03-04T16:17:00.516 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.516 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.516 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.516 DEBUG:boto:Params: {} 2024-03-04T16:17:00.517 DEBUG:boto:Token: None 2024-03-04T16:17:00.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/?versions 2024-03-04T16:17:00.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk= 2024-03-04T16:17:00.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk=', 'Content-Length': '0'} 2024-03-04T16:17:00.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc4c695a282a61e-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.519 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-03-04T16:17:00.520 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-03-04T16:17:00.521 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-03-04T16:17:00.521 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-03-04T16:17:00.521 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:00.521 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.521 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.521 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.521 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.521 DEBUG:boto:Method: GET 2024-03-04T16:17:00.521 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.521 DEBUG:boto:Data: 2024-03-04T16:17:00.521 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.522 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.522 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.522 DEBUG:boto:Params: {} 2024-03-04T16:17:00.522 DEBUG:boto:Token: None 2024-03-04T16:17:00.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/?versions 2024-03-04T16:17:00.522 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk= 2024-03-04T16:17:00.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk=', 'Content-Length': '0'} 2024-03-04T16:17:00.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002407bde2606ac21-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.524 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.525 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.525 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.525 DEBUG:boto:path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.525 DEBUG:boto:auth_path=/sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.525 DEBUG:boto:Method: GET 2024-03-04T16:17:00.526 DEBUG:boto:Path: /sbnvju-12/?versions&max-keys=999 2024-03-04T16:17:00.526 DEBUG:boto:Data: 2024-03-04T16:17:00.526 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.526 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.526 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.526 DEBUG:boto:Params: {} 2024-03-04T16:17:00.526 DEBUG:boto:Token: None 2024-03-04T16:17:00.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/?versions 2024-03-04T16:17:00.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk= 2024-03-04T16:17:00.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Cnmj26EnbWlDMZhrfIoaIcP2Pmk=', 'Content-Length': '0'} 2024-03-04T16:17:00.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f8ce9377a139506-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.529 DEBUG:boto:b'sbnvju-12999false&nulltrue2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.nulltrue2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..nulltrue2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...nulltrue2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.onulltrue2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.nulltrue2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:17:00.529 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.529 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.530 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.530 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.530 DEBUG:boto:Method: GET 2024-03-04T16:17:00.530 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.530 DEBUG:boto:Data: 2024-03-04T16:17:00.530 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.530 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.530 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.530 DEBUG:boto:Params: {} 2024-03-04T16:17:00.530 DEBUG:boto:Token: None 2024-03-04T16:17:00.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs= 2024-03-04T16:17:00.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs=', 'Content-Length': '0'} 2024-03-04T16:17:00.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2622ce82d5e66d0-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.532 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.533 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.533 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.533 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.533 DEBUG:boto:Method: GET 2024-03-04T16:17:00.533 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.533 DEBUG:boto:Data: 2024-03-04T16:17:00.533 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.533 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.533 DEBUG:boto:Params: {} 2024-03-04T16:17:00.533 DEBUG:boto:Token: None 2024-03-04T16:17:00.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs= 2024-03-04T16:17:00.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs=', 'Content-Length': '0'} 2024-03-04T16:17:00.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef0ece342dcffe60-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.536 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.536 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.536 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.536 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.536 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.536 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.536 DEBUG:boto:Data: 2024-03-04T16:17:00.536 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.536 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.536 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.536 DEBUG:boto:Params: {} 2024-03-04T16:17:00.537 DEBUG:boto:Token: None 2024-03-04T16:17:00.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.537 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+QPYLlj/0TwycP7ZhnTq5Xzvjs= 2024-03-04T16:17:00.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+QPYLlj/0TwycP7ZhnTq5Xzvjs=', 'Content-Length': '0'} 2024-03-04T16:17:00.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000208f6bce4da0383d-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.538 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.539 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.539 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.539 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.539 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.539 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.539 DEBUG:boto:Data: 2024-03-04T16:17:00.539 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.539 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.539 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.539 DEBUG:boto:Params: {} 2024-03-04T16:17:00.540 DEBUG:boto:Token: None 2024-03-04T16:17:00.540 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.540 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+QPYLlj/0TwycP7ZhnTq5Xzvjs= 2024-03-04T16:17:00.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+QPYLlj/0TwycP7ZhnTq5Xzvjs=', 'Content-Length': '0'} 2024-03-04T16:17:00.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052214d06c80c33dd-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:17:00.542 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.542 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.542 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.542 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.542 DEBUG:boto:Method: GET 2024-03-04T16:17:00.542 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.542 DEBUG:boto:Data: 2024-03-04T16:17:00.542 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.542 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.542 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.543 DEBUG:boto:Params: {} 2024-03-04T16:17:00.543 DEBUG:boto:Token: None 2024-03-04T16:17:00.543 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs= 2024-03-04T16:17:00.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs=', 'Content-Length': '0'} 2024-03-04T16:17:00.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047884f11edde612c-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.545 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.545 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.545 DEBUG:boto:path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.545 DEBUG:boto:auth_path=/sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.545 DEBUG:boto:Method: GET 2024-03-04T16:17:00.545 DEBUG:boto:Path: /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.545 DEBUG:boto:Data: 2024-03-04T16:17:00.545 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.546 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.546 DEBUG:boto:Params: {} 2024-03-04T16:17:00.546 DEBUG:boto:Token: None 2024-03-04T16:17:00.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26?versionId=null 2024-03-04T16:17:00.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs= 2024-03-04T16:17:00.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ATRI3Z2CgAOMXK5TlwU+3D6Yhs=', 'Content-Length': '0'} 2024-03-04T16:17:00.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e534826f1ff03e53-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.548 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.548 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.548 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.548 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.548 DEBUG:boto:Method: GET 2024-03-04T16:17:00.548 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:17:00.548 DEBUG:boto:Data: 2024-03-04T16:17:00.548 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.549 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.549 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.549 DEBUG:boto:Params: {} 2024-03-04T16:17:00.549 DEBUG:boto:Token: None 2024-03-04T16:17:00.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db40dfac9339a065-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.592 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.592 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.592 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.593 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.593 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.593 DEBUG:boto:Method: GET 2024-03-04T16:17:00.593 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:17:00.593 DEBUG:boto:Data: 2024-03-04T16:17:00.593 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.593 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.593 DEBUG:boto:Params: {} 2024-03-04T16:17:00.593 DEBUG:boto:Token: None 2024-03-04T16:17:00.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa8d932dbf5a7cd-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.596 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.596 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.596 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.596 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.596 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:17:00.597 DEBUG:boto:Data: 2024-03-04T16:17:00.597 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.597 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.597 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.597 DEBUG:boto:Params: {} 2024-03-04T16:17:00.597 DEBUG:boto:Token: None 2024-03-04T16:17:00.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26 2024-03-04T16:17:00.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xJ+3Aoj4gHJeLAgd/0f0NZvLPgM= 2024-03-04T16:17:00.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xJ+3Aoj4gHJeLAgd/0f0NZvLPgM=', 'Content-Length': '0'} 2024-03-04T16:17:00.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036108f0d99fdfd27-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.599 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.599 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.599 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.599 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:17:00.599 DEBUG:boto:Data: 2024-03-04T16:17:00.599 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.600 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.600 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.600 DEBUG:boto:Params: {} 2024-03-04T16:17:00.600 DEBUG:boto:Token: None 2024-03-04T16:17:00.600 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26 2024-03-04T16:17:00.600 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xJ+3Aoj4gHJeLAgd/0f0NZvLPgM= 2024-03-04T16:17:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xJ+3Aoj4gHJeLAgd/0f0NZvLPgM=', 'Content-Length': '0'} 2024-03-04T16:17:00.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097fc591390892193-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-03-04T16:17:00.602 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.602 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.602 DEBUG:boto:Method: GET 2024-03-04T16:17:00.602 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:17:00.602 DEBUG:boto:Data: 2024-03-04T16:17:00.602 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.603 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.603 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.603 DEBUG:boto:Params: {} 2024-03-04T16:17:00.603 DEBUG:boto:Token: None 2024-03-04T16:17:00.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26 2024-03-04T16:17:00.603 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zl5SZuJByROJtrSQ0KisCElsE4A= 2024-03-04T16:17:00.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zl5SZuJByROJtrSQ0KisCElsE4A=', 'Content-Length': '0'} 2024-03-04T16:17:00.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a20fe4dafac2b78-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.605 DEBUG:boto:path=/sbnvju-12/%26 2024-03-04T16:17:00.605 DEBUG:boto:auth_path=/sbnvju-12/%26 2024-03-04T16:17:00.605 DEBUG:boto:Method: GET 2024-03-04T16:17:00.605 DEBUG:boto:Path: /sbnvju-12/%26 2024-03-04T16:17:00.605 DEBUG:boto:Data: 2024-03-04T16:17:00.605 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.606 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.606 DEBUG:boto:Params: {} 2024-03-04T16:17:00.606 DEBUG:boto:Token: None 2024-03-04T16:17:00.606 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/%26 2024-03-04T16:17:00.606 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zl5SZuJByROJtrSQ0KisCElsE4A= 2024-03-04T16:17:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zl5SZuJByROJtrSQ0KisCElsE4A=', 'Content-Length': '0'} 2024-03-04T16:17:00.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc7b181438eae612-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.608 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.608 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.608 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.609 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.609 DEBUG:boto:Method: GET 2024-03-04T16:17:00.609 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:17:00.609 DEBUG:boto:Data: 2024-03-04T16:17:00.609 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.609 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.609 DEBUG:boto:Params: {} 2024-03-04T16:17:00.609 DEBUG:boto:Token: None 2024-03-04T16:17:00.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.609 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ea3f1593fe846e-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.612 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.612 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.612 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.612 DEBUG:boto:path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.612 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%26 2024-03-04T16:17:00.613 DEBUG:boto:Method: GET 2024-03-04T16:17:00.613 DEBUG:boto:Path: /sbnvju-12/?prefix=%26 2024-03-04T16:17:00.613 DEBUG:boto:Data: 2024-03-04T16:17:00.613 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.613 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.613 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.613 DEBUG:boto:Params: {} 2024-03-04T16:17:00.613 DEBUG:boto:Token: None 2024-03-04T16:17:00.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.613 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f789f34d2578cd8-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.616 DEBUG:boto:b'sbnvju-12&1000false&2024-03-04T16:13:01.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:17:00.616 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.616 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.616 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.617 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.617 DEBUG:boto:Method: GET 2024-03-04T16:17:00.617 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.617 DEBUG:boto:Data: 2024-03-04T16:17:00.617 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.617 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.617 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.617 DEBUG:boto:Params: {} 2024-03-04T16:17:00.617 DEBUG:boto:Token: None 2024-03-04T16:17:00.617 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.617 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI= 2024-03-04T16:17:00.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI=', 'Content-Length': '0'} 2024-03-04T16:17:00.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e15a0b2037c4c6e-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.619 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.619 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.619 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.620 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.620 DEBUG:boto:Method: GET 2024-03-04T16:17:00.620 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.620 DEBUG:boto:Data: 2024-03-04T16:17:00.620 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.620 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.620 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.620 DEBUG:boto:Params: {} 2024-03-04T16:17:00.620 DEBUG:boto:Token: None 2024-03-04T16:17:00.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.621 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI= 2024-03-04T16:17:00.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI=', 'Content-Length': '0'} 2024-03-04T16:17:00.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016b47c98ebeea82a-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.623 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.623 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.623 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.623 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.623 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.623 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.623 DEBUG:boto:Data: 2024-03-04T16:17:00.623 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.623 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.624 DEBUG:boto:Params: {} 2024-03-04T16:17:00.624 DEBUG:boto:Token: None 2024-03-04T16:17:00.624 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.624 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kEH+ycKETHf+LwvfyOTigNsHP1Y= 2024-03-04T16:17:00.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kEH+ycKETHf+LwvfyOTigNsHP1Y=', 'Content-Length': '0'} 2024-03-04T16:17:00.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f6c72e4d425298c-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.626 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.626 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.626 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.626 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.626 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.626 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.626 DEBUG:boto:Data: 2024-03-04T16:17:00.626 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.626 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.627 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.627 DEBUG:boto:Params: {} 2024-03-04T16:17:00.627 DEBUG:boto:Token: None 2024-03-04T16:17:00.627 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.627 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kEH+ycKETHf+LwvfyOTigNsHP1Y= 2024-03-04T16:17:00.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kEH+ycKETHf+LwvfyOTigNsHP1Y=', 'Content-Length': '0'} 2024-03-04T16:17:00.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a33b7d38a06c759-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:17:00.629 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.629 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.629 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.629 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.629 DEBUG:boto:Method: GET 2024-03-04T16:17:00.629 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.629 DEBUG:boto:Data: 2024-03-04T16:17:00.629 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.630 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.630 DEBUG:boto:Params: {} 2024-03-04T16:17:00.630 DEBUG:boto:Token: None 2024-03-04T16:17:00.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI= 2024-03-04T16:17:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI=', 'Content-Length': '0'} 2024-03-04T16:17:00.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daf5cfb5945b9db2-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.632 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.632 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.632 DEBUG:boto:path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.632 DEBUG:boto:auth_path=/sbnvju-12/.?versionId=null 2024-03-04T16:17:00.632 DEBUG:boto:Method: GET 2024-03-04T16:17:00.632 DEBUG:boto:Path: /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.632 DEBUG:boto:Data: 2024-03-04T16:17:00.632 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.633 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.633 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.633 DEBUG:boto:Params: {} 2024-03-04T16:17:00.633 DEBUG:boto:Token: None 2024-03-04T16:17:00.633 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.?versionId=null 2024-03-04T16:17:00.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI= 2024-03-04T16:17:00.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gv5p1j/5vjsWOZxsDTs+LftE9gI=', 'Content-Length': '0'} 2024-03-04T16:17:00.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000547d6d7a1b487086-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.676 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.676 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.677 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.677 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.677 DEBUG:boto:Method: GET 2024-03-04T16:17:00.677 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:17:00.677 DEBUG:boto:Data: 2024-03-04T16:17:00.677 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.677 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.677 DEBUG:boto:Params: {} 2024-03-04T16:17:00.677 DEBUG:boto:Token: None 2024-03-04T16:17:00.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008646bb89026ba5e0-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.680 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.681 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.681 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.681 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.681 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.681 DEBUG:boto:Method: GET 2024-03-04T16:17:00.681 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:17:00.681 DEBUG:boto:Data: 2024-03-04T16:17:00.681 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.681 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.681 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.681 DEBUG:boto:Params: {} 2024-03-04T16:17:00.682 DEBUG:boto:Token: None 2024-03-04T16:17:00.682 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.682 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d7939980d85160f-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.684 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.685 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.685 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.685 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.685 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:17:00.685 DEBUG:boto:Data: 2024-03-04T16:17:00.685 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.686 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.686 DEBUG:boto:Params: {} 2024-03-04T16:17:00.686 DEBUG:boto:Token: None 2024-03-04T16:17:00.686 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/. 2024-03-04T16:17:00.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xDSX1Fjx9Ep/xHS+9rgokPiWp94= 2024-03-04T16:17:00.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xDSX1Fjx9Ep/xHS+9rgokPiWp94=', 'Content-Length': '0'} 2024-03-04T16:17:00.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010fbd542c6ed4170-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.688 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.688 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.688 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.688 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:17:00.688 DEBUG:boto:Data: 2024-03-04T16:17:00.688 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.689 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.689 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.689 DEBUG:boto:Params: {} 2024-03-04T16:17:00.689 DEBUG:boto:Token: None 2024-03-04T16:17:00.689 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/. 2024-03-04T16:17:00.689 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xDSX1Fjx9Ep/xHS+9rgokPiWp94= 2024-03-04T16:17:00.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xDSX1Fjx9Ep/xHS+9rgokPiWp94=', 'Content-Length': '0'} 2024-03-04T16:17:00.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b66831660b6841c6-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-03-04T16:17:00.691 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.691 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.691 DEBUG:boto:Method: GET 2024-03-04T16:17:00.691 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:17:00.691 DEBUG:boto:Data: 2024-03-04T16:17:00.691 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.691 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.691 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.691 DEBUG:boto:Params: {} 2024-03-04T16:17:00.692 DEBUG:boto:Token: None 2024-03-04T16:17:00.692 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/. 2024-03-04T16:17:00.692 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7tGHYcpEqTwRnBpwIEjLlG/5Fko= 2024-03-04T16:17:00.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7tGHYcpEqTwRnBpwIEjLlG/5Fko=', 'Content-Length': '0'} 2024-03-04T16:17:00.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3a9dc55dadc7856-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.736 DEBUG:boto:path=/sbnvju-12/. 2024-03-04T16:17:00.736 DEBUG:boto:auth_path=/sbnvju-12/. 2024-03-04T16:17:00.736 DEBUG:boto:Method: GET 2024-03-04T16:17:00.736 DEBUG:boto:Path: /sbnvju-12/. 2024-03-04T16:17:00.736 DEBUG:boto:Data: 2024-03-04T16:17:00.736 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.737 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.737 DEBUG:boto:Params: {} 2024-03-04T16:17:00.737 DEBUG:boto:Token: None 2024-03-04T16:17:00.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/. 2024-03-04T16:17:00.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7tGHYcpEqTwRnBpwIEjLlG/5Fko= 2024-03-04T16:17:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7tGHYcpEqTwRnBpwIEjLlG/5Fko=', 'Content-Length': '0'} 2024-03-04T16:17:00.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000712e0368c30d73b0-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.739 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.739 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.739 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.739 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.739 DEBUG:boto:Method: GET 2024-03-04T16:17:00.739 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:17:00.739 DEBUG:boto:Data: 2024-03-04T16:17:00.739 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.739 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.740 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.740 DEBUG:boto:Params: {} 2024-03-04T16:17:00.740 DEBUG:boto:Token: None 2024-03-04T16:17:00.740 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.740 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063fed8b450906ed7-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.742 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.743 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.743 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.743 DEBUG:boto:path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.743 DEBUG:boto:auth_path=/sbnvju-12/?prefix=. 2024-03-04T16:17:00.743 DEBUG:boto:Method: GET 2024-03-04T16:17:00.743 DEBUG:boto:Path: /sbnvju-12/?prefix=. 2024-03-04T16:17:00.743 DEBUG:boto:Data: 2024-03-04T16:17:00.743 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.743 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.744 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.744 DEBUG:boto:Params: {} 2024-03-04T16:17:00.744 DEBUG:boto:Token: None 2024-03-04T16:17:00.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.744 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032767741b545b47d-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.746 DEBUG:boto:b'sbnvju-12.1000false.2024-03-04T16:13:01.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:17:00.747 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.747 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.747 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.747 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.747 DEBUG:boto:Method: GET 2024-03-04T16:17:00.747 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.747 DEBUG:boto:Data: 2024-03-04T16:17:00.747 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.748 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.748 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.748 DEBUG:boto:Params: {} 2024-03-04T16:17:00.748 DEBUG:boto:Token: None 2024-03-04T16:17:00.748 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk= 2024-03-04T16:17:00.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk=', 'Content-Length': '0'} 2024-03-04T16:17:00.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7000eaa2d88e9cf-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.750 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.750 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.750 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.750 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.751 DEBUG:boto:Method: GET 2024-03-04T16:17:00.751 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.751 DEBUG:boto:Data: 2024-03-04T16:17:00.751 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.751 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.751 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.751 DEBUG:boto:Params: {} 2024-03-04T16:17:00.751 DEBUG:boto:Token: None 2024-03-04T16:17:00.751 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.751 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk= 2024-03-04T16:17:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk=', 'Content-Length': '0'} 2024-03-04T16:17:00.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ac5bf0aedec374a-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.753 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.753 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.754 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.754 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.754 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.754 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.754 DEBUG:boto:Data: 2024-03-04T16:17:00.754 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.754 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.754 DEBUG:boto:Params: {} 2024-03-04T16:17:00.754 DEBUG:boto:Token: None 2024-03-04T16:17:00.754 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f+YrLM45ez9iqn3cr+JcQK1fbFc= 2024-03-04T16:17:00.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f+YrLM45ez9iqn3cr+JcQK1fbFc=', 'Content-Length': '0'} 2024-03-04T16:17:00.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323fe3cfb80c00f7-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.756 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.756 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.757 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.757 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.757 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.757 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.757 DEBUG:boto:Data: 2024-03-04T16:17:00.757 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.757 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.757 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.757 DEBUG:boto:Params: {} 2024-03-04T16:17:00.757 DEBUG:boto:Token: None 2024-03-04T16:17:00.757 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.757 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f+YrLM45ez9iqn3cr+JcQK1fbFc= 2024-03-04T16:17:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f+YrLM45ez9iqn3cr+JcQK1fbFc=', 'Content-Length': '0'} 2024-03-04T16:17:00.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a37435d23724b98d-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:17:00.759 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.759 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.759 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.760 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.760 DEBUG:boto:Method: GET 2024-03-04T16:17:00.760 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.760 DEBUG:boto:Data: 2024-03-04T16:17:00.760 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.760 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.760 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.760 DEBUG:boto:Params: {} 2024-03-04T16:17:00.760 DEBUG:boto:Token: None 2024-03-04T16:17:00.760 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.760 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk= 2024-03-04T16:17:00.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk=', 'Content-Length': '0'} 2024-03-04T16:17:00.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ea4f3522432a2f7-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.762 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.762 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.762 DEBUG:boto:path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.763 DEBUG:boto:auth_path=/sbnvju-12/..?versionId=null 2024-03-04T16:17:00.763 DEBUG:boto:Method: GET 2024-03-04T16:17:00.763 DEBUG:boto:Path: /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.763 DEBUG:boto:Data: 2024-03-04T16:17:00.763 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.763 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.763 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.763 DEBUG:boto:Params: {} 2024-03-04T16:17:00.763 DEBUG:boto:Token: None 2024-03-04T16:17:00.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/..?versionId=null 2024-03-04T16:17:00.763 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk= 2024-03-04T16:17:00.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D1H2TAL3QxekzH6GWhBQM2c2Upk=', 'Content-Length': '0'} 2024-03-04T16:17:00.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f22d2ada9dfffd34-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.765 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.765 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.766 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.766 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.766 DEBUG:boto:Method: GET 2024-03-04T16:17:00.766 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:17:00.766 DEBUG:boto:Data: 2024-03-04T16:17:00.766 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.766 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.766 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.766 DEBUG:boto:Params: {} 2024-03-04T16:17:00.766 DEBUG:boto:Token: None 2024-03-04T16:17:00.766 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.767 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d72d32fc542fbbb-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.769 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.769 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.769 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.769 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.769 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.769 DEBUG:boto:Method: GET 2024-03-04T16:17:00.769 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:17:00.770 DEBUG:boto:Data: 2024-03-04T16:17:00.770 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.770 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.770 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.770 DEBUG:boto:Params: {} 2024-03-04T16:17:00.770 DEBUG:boto:Token: None 2024-03-04T16:17:00.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.770 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000382c3c845fc208b7-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.772 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.773 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.773 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.773 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.773 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:17:00.773 DEBUG:boto:Data: 2024-03-04T16:17:00.773 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.773 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.773 DEBUG:boto:Params: {} 2024-03-04T16:17:00.774 DEBUG:boto:Token: None 2024-03-04T16:17:00.774 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.. 2024-03-04T16:17:00.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AQgbQsky3WW+yNSJxr62Sxv7PXg= 2024-03-04T16:17:00.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AQgbQsky3WW+yNSJxr62Sxv7PXg=', 'Content-Length': '0'} 2024-03-04T16:17:00.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aba5b0c76770a5c-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.776 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.776 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.776 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.776 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:17:00.776 DEBUG:boto:Data: 2024-03-04T16:17:00.776 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.776 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.776 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.776 DEBUG:boto:Params: {} 2024-03-04T16:17:00.776 DEBUG:boto:Token: None 2024-03-04T16:17:00.776 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.. 2024-03-04T16:17:00.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AQgbQsky3WW+yNSJxr62Sxv7PXg= 2024-03-04T16:17:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AQgbQsky3WW+yNSJxr62Sxv7PXg=', 'Content-Length': '0'} 2024-03-04T16:17:00.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7f8e6d60c29cb4-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-03-04T16:17:00.778 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.779 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.779 DEBUG:boto:Method: GET 2024-03-04T16:17:00.779 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:17:00.779 DEBUG:boto:Data: 2024-03-04T16:17:00.779 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.779 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.779 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.779 DEBUG:boto:Params: {} 2024-03-04T16:17:00.779 DEBUG:boto:Token: None 2024-03-04T16:17:00.779 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.. 2024-03-04T16:17:00.779 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uaTsbUAYvOslxcfKi4irn1ibzBE= 2024-03-04T16:17:00.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uaTsbUAYvOslxcfKi4irn1ibzBE=', 'Content-Length': '0'} 2024-03-04T16:17:00.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c25a1ff6cc6bb898-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.824 DEBUG:boto:path=/sbnvju-12/.. 2024-03-04T16:17:00.824 DEBUG:boto:auth_path=/sbnvju-12/.. 2024-03-04T16:17:00.824 DEBUG:boto:Method: GET 2024-03-04T16:17:00.824 DEBUG:boto:Path: /sbnvju-12/.. 2024-03-04T16:17:00.825 DEBUG:boto:Data: 2024-03-04T16:17:00.825 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.825 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.825 DEBUG:boto:Params: {} 2024-03-04T16:17:00.825 DEBUG:boto:Token: None 2024-03-04T16:17:00.825 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.. 2024-03-04T16:17:00.825 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uaTsbUAYvOslxcfKi4irn1ibzBE= 2024-03-04T16:17:00.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uaTsbUAYvOslxcfKi4irn1ibzBE=', 'Content-Length': '0'} 2024-03-04T16:17:00.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b30ed3745880406-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.827 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.827 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.828 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.828 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.828 DEBUG:boto:Method: GET 2024-03-04T16:17:00.828 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:17:00.828 DEBUG:boto:Data: 2024-03-04T16:17:00.828 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.828 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.828 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.828 DEBUG:boto:Params: {} 2024-03-04T16:17:00.828 DEBUG:boto:Token: None 2024-03-04T16:17:00.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5893c2773770570-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.831 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.831 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.831 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.831 DEBUG:boto:path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.832 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.. 2024-03-04T16:17:00.832 DEBUG:boto:Method: GET 2024-03-04T16:17:00.832 DEBUG:boto:Path: /sbnvju-12/?prefix=.. 2024-03-04T16:17:00.832 DEBUG:boto:Data: 2024-03-04T16:17:00.832 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.832 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.832 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.832 DEBUG:boto:Params: {} 2024-03-04T16:17:00.832 DEBUG:boto:Token: None 2024-03-04T16:17:00.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084cc323d60be1063-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.835 DEBUG:boto:b'sbnvju-12..1000false..2024-03-04T16:13:01.738Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:17:00.836 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.836 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.836 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.836 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.836 DEBUG:boto:Method: GET 2024-03-04T16:17:00.837 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.837 DEBUG:boto:Data: 2024-03-04T16:17:00.837 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.837 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.837 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.837 DEBUG:boto:Params: {} 2024-03-04T16:17:00.837 DEBUG:boto:Token: None 2024-03-04T16:17:00.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI= 2024-03-04T16:17:00.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI=', 'Content-Length': '0'} 2024-03-04T16:17:00.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c47e7f9507dd7a1c-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.839 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.839 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.839 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.839 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.840 DEBUG:boto:Method: GET 2024-03-04T16:17:00.840 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.840 DEBUG:boto:Data: 2024-03-04T16:17:00.840 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.840 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.840 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.840 DEBUG:boto:Params: {} 2024-03-04T16:17:00.840 DEBUG:boto:Token: None 2024-03-04T16:17:00.840 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.840 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI= 2024-03-04T16:17:00.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI=', 'Content-Length': '0'} 2024-03-04T16:17:00.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008211730706ea25fb-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.842 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.842 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.842 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.843 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.843 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.843 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.843 DEBUG:boto:Data: 2024-03-04T16:17:00.843 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.843 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.843 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.843 DEBUG:boto:Params: {} 2024-03-04T16:17:00.843 DEBUG:boto:Token: None 2024-03-04T16:17:00.843 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.843 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6e01a2YL/Qr0oXwGiTPGBDlGt8= 2024-03-04T16:17:00.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6e01a2YL/Qr0oXwGiTPGBDlGt8=', 'Content-Length': '0'} 2024-03-04T16:17:00.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cee43bbc98927ab-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.845 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.845 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.846 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.846 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.846 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.846 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.846 DEBUG:boto:Data: 2024-03-04T16:17:00.846 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.846 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.846 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.846 DEBUG:boto:Params: {} 2024-03-04T16:17:00.846 DEBUG:boto:Token: None 2024-03-04T16:17:00.846 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6e01a2YL/Qr0oXwGiTPGBDlGt8= 2024-03-04T16:17:00.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6e01a2YL/Qr0oXwGiTPGBDlGt8=', 'Content-Length': '0'} 2024-03-04T16:17:00.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2ed52f50a1114b9-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:17:00.849 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.849 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.849 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.849 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.849 DEBUG:boto:Method: GET 2024-03-04T16:17:00.849 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.849 DEBUG:boto:Data: 2024-03-04T16:17:00.849 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.849 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.849 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.850 DEBUG:boto:Params: {} 2024-03-04T16:17:00.850 DEBUG:boto:Token: None 2024-03-04T16:17:00.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI= 2024-03-04T16:17:00.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI=', 'Content-Length': '0'} 2024-03-04T16:17:00.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a31d7ef53e8b8b8-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.852 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.852 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.852 DEBUG:boto:path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.852 DEBUG:boto:auth_path=/sbnvju-12/...?versionId=null 2024-03-04T16:17:00.852 DEBUG:boto:Method: GET 2024-03-04T16:17:00.852 DEBUG:boto:Path: /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.852 DEBUG:boto:Data: 2024-03-04T16:17:00.852 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.853 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.853 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.853 DEBUG:boto:Params: {} 2024-03-04T16:17:00.853 DEBUG:boto:Token: None 2024-03-04T16:17:00.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/...?versionId=null 2024-03-04T16:17:00.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI= 2024-03-04T16:17:00.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DjeG8yoWUjgeThJq2ACgwUP2PYI=', 'Content-Length': '0'} 2024-03-04T16:17:00.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a63cba6d20fc067-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.855 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.855 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.855 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.855 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.855 DEBUG:boto:Method: GET 2024-03-04T16:17:00.855 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:17:00.855 DEBUG:boto:Data: 2024-03-04T16:17:00.855 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.856 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.856 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.856 DEBUG:boto:Params: {} 2024-03-04T16:17:00.856 DEBUG:boto:Token: None 2024-03-04T16:17:00.856 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.856 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c6420e811f37d8-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.859 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.859 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.859 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.859 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.859 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.860 DEBUG:boto:Method: GET 2024-03-04T16:17:00.860 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:17:00.860 DEBUG:boto:Data: 2024-03-04T16:17:00.860 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.860 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.861 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.861 DEBUG:boto:Params: {} 2024-03-04T16:17:00.861 DEBUG:boto:Token: None 2024-03-04T16:17:00.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c759f676a926942-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.864 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.865 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.865 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.865 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.865 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:17:00.865 DEBUG:boto:Data: 2024-03-04T16:17:00.866 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.866 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.866 DEBUG:boto:Params: {} 2024-03-04T16:17:00.866 DEBUG:boto:Token: None 2024-03-04T16:17:00.866 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/... 2024-03-04T16:17:00.867 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vO52ICbgF1jjhAxJMbM6OVYSkVE= 2024-03-04T16:17:00.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vO52ICbgF1jjhAxJMbM6OVYSkVE=', 'Content-Length': '0'} 2024-03-04T16:17:00.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095eba6bfec74d8d2-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.869 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.869 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.870 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.870 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:17:00.870 DEBUG:boto:Data: 2024-03-04T16:17:00.870 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.870 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.870 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.870 DEBUG:boto:Params: {} 2024-03-04T16:17:00.871 DEBUG:boto:Token: None 2024-03-04T16:17:00.871 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/... 2024-03-04T16:17:00.871 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vO52ICbgF1jjhAxJMbM6OVYSkVE= 2024-03-04T16:17:00.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vO52ICbgF1jjhAxJMbM6OVYSkVE=', 'Content-Length': '0'} 2024-03-04T16:17:00.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074529728e33b6c69-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-03-04T16:17:00.876 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.877 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.877 DEBUG:boto:Method: GET 2024-03-04T16:17:00.877 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:17:00.877 DEBUG:boto:Data: 2024-03-04T16:17:00.877 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.877 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.877 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.878 DEBUG:boto:Params: {} 2024-03-04T16:17:00.878 DEBUG:boto:Token: None 2024-03-04T16:17:00.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/... 2024-03-04T16:17:00.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MWdzr3gDh816UnjQbnGzhSrVRQA= 2024-03-04T16:17:00.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MWdzr3gDh816UnjQbnGzhSrVRQA=', 'Content-Length': '0'} 2024-03-04T16:17:00.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cb8f30971d6898b-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.881 DEBUG:boto:path=/sbnvju-12/... 2024-03-04T16:17:00.881 DEBUG:boto:auth_path=/sbnvju-12/... 2024-03-04T16:17:00.881 DEBUG:boto:Method: GET 2024-03-04T16:17:00.881 DEBUG:boto:Path: /sbnvju-12/... 2024-03-04T16:17:00.881 DEBUG:boto:Data: 2024-03-04T16:17:00.881 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.882 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.882 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.882 DEBUG:boto:Params: {} 2024-03-04T16:17:00.882 DEBUG:boto:Token: None 2024-03-04T16:17:00.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/... 2024-03-04T16:17:00.883 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MWdzr3gDh816UnjQbnGzhSrVRQA= 2024-03-04T16:17:00.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MWdzr3gDh816UnjQbnGzhSrVRQA=', 'Content-Length': '0'} 2024-03-04T16:17:00.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043490cc454a74844-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.928 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.928 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.928 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.929 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.929 DEBUG:boto:Method: GET 2024-03-04T16:17:00.929 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:17:00.929 DEBUG:boto:Data: 2024-03-04T16:17:00.929 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.930 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.930 DEBUG:boto:Params: {} 2024-03-04T16:17:00.930 DEBUG:boto:Token: None 2024-03-04T16:17:00.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018e0275c64f01167-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.933 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.934 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:00.934 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:00.934 DEBUG:boto:path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.934 DEBUG:boto:auth_path=/sbnvju-12/?prefix=... 2024-03-04T16:17:00.934 DEBUG:boto:Method: GET 2024-03-04T16:17:00.934 DEBUG:boto:Path: /sbnvju-12/?prefix=... 2024-03-04T16:17:00.935 DEBUG:boto:Data: 2024-03-04T16:17:00.935 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.935 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.935 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.935 DEBUG:boto:Params: {} 2024-03-04T16:17:00.935 DEBUG:boto:Token: None 2024-03-04T16:17:00.936 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/ 2024-03-04T16:17:00.936 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg= 2024-03-04T16:17:00.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QRGomFvlWLWPWbkrFSDwZ1+xzWg=', 'Content-Length': '0'} 2024-03-04T16:17:00.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c712d96830b66cbf-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.938 DEBUG:boto:b'sbnvju-12...1000false...2024-03-04T16:13:01.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:00.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:17:00.939 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.939 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.939 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.940 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.940 DEBUG:boto:Method: GET 2024-03-04T16:17:00.940 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.940 DEBUG:boto:Data: 2024-03-04T16:17:00.940 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.940 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.941 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.941 DEBUG:boto:Params: {} 2024-03-04T16:17:00.941 DEBUG:boto:Token: None 2024-03-04T16:17:00.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.941 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA= 2024-03-04T16:17:00.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA=', 'Content-Length': '0'} 2024-03-04T16:17:00.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb9117c9f7200bf9-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.984 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.984 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.984 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.984 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.984 DEBUG:boto:Method: GET 2024-03-04T16:17:00.985 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.985 DEBUG:boto:Data: 2024-03-04T16:17:00.985 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.985 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.985 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.985 DEBUG:boto:Params: {} 2024-03-04T16:17:00.985 DEBUG:boto:Token: None 2024-03-04T16:17:00.985 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.985 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA= 2024-03-04T16:17:00.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA=', 'Content-Length': '0'} 2024-03-04T16:17:00.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070386a064e91b4bb-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.988 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.988 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.988 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.988 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.988 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.988 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.989 DEBUG:boto:Data: 2024-03-04T16:17:00.989 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.989 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.989 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.989 DEBUG:boto:Params: {} 2024-03-04T16:17:00.989 DEBUG:boto:Token: None 2024-03-04T16:17:00.989 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.989 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:54UQ0fqFARHuyQvX9U12GWsh58A= 2024-03-04T16:17:00.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:54UQ0fqFARHuyQvX9U12GWsh58A=', 'Content-Length': '0'} 2024-03-04T16:17:00.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000545f48b525c183c-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.991 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.991 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.991 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.991 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.991 DEBUG:boto:Method: HEAD 2024-03-04T16:17:00.992 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.992 DEBUG:boto:Data: 2024-03-04T16:17:00.992 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.992 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:00.992 DEBUG:boto:Port: 8001 2024-03-04T16:17:00.992 DEBUG:boto:Params: {} 2024-03-04T16:17:00.992 DEBUG:boto:Token: None 2024-03-04T16:17:00.992 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.992 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:54UQ0fqFARHuyQvX9U12GWsh58A= 2024-03-04T16:17:00.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:54UQ0fqFARHuyQvX9U12GWsh58A=', 'Content-Length': '0'} 2024-03-04T16:17:00.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062ab0d080d1e89c3-0065e5f3fc-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:17:00.995 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.995 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.995 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.995 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.995 DEBUG:boto:Method: GET 2024-03-04T16:17:00.995 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.995 DEBUG:boto:Data: 2024-03-04T16:17:00.995 DEBUG:boto:Headers: {} 2024-03-04T16:17:00.996 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:00.996 DEBUG:boto:Port: 8000 2024-03-04T16:17:00.996 DEBUG:boto:Params: {} 2024-03-04T16:17:00.996 DEBUG:boto:Token: None 2024-03-04T16:17:00.996 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:00 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.997 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA= 2024-03-04T16:17:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q4/fX170C2RIu3MSF/JfwZRq2JA=', 'Content-Length': '0'} 2024-03-04T16:17:00.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000528a9b9cc9b430f-0065e5f3fc-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:00.999 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:00.999 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:00.999 DEBUG:boto:path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:00.999 DEBUG:boto:auth_path=/sbnvju-12/.o?versionId=null 2024-03-04T16:17:01.000 DEBUG:boto:Method: GET 2024-03-04T16:17:01.000 DEBUG:boto:Path: /sbnvju-12/.o?versionId=null 2024-03-04T16:17:01.000 DEBUG:boto:Data: 2024-03-04T16:17:01.000 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.000 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.000 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.000 DEBUG:boto:Params: {} 2024-03-04T16:17:01.000 DEBUG:boto:Token: None 2024-03-04T16:17:01.000 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o?versionId=null 2024-03-04T16:17:01.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fiPVdEtgdH7WTsrzsobQxj6qQgY= 2024-03-04T16:17:01.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fiPVdEtgdH7WTsrzsobQxj6qQgY=', 'Content-Length': '0'} 2024-03-04T16:17:01.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e05a7bad278f690-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.002 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.003 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.003 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.003 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.003 DEBUG:boto:Method: GET 2024-03-04T16:17:01.003 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:17:01.003 DEBUG:boto:Data: 2024-03-04T16:17:01.003 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.003 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.003 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.003 DEBUG:boto:Params: {} 2024-03-04T16:17:01.004 DEBUG:boto:Token: None 2024-03-04T16:17:01.004 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.004 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2aab1665e212d86-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.006 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.007 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.007 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.007 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.007 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.007 DEBUG:boto:Method: GET 2024-03-04T16:17:01.007 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:17:01.007 DEBUG:boto:Data: 2024-03-04T16:17:01.007 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.007 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.007 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.007 DEBUG:boto:Params: {} 2024-03-04T16:17:01.007 DEBUG:boto:Token: None 2024-03-04T16:17:01.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.008 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a745bc9a89ae4ced-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.011 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.011 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:01.012 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:01.012 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.012 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:17:01.012 DEBUG:boto:Data: 2024-03-04T16:17:01.012 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.012 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.012 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.012 DEBUG:boto:Params: {} 2024-03-04T16:17:01.012 DEBUG:boto:Token: None 2024-03-04T16:17:01.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o 2024-03-04T16:17:01.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d895qPjW/q1aUyDT5OX9dbubdAg= 2024-03-04T16:17:01.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d895qPjW/q1aUyDT5OX9dbubdAg=', 'Content-Length': '0'} 2024-03-04T16:17:01.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012adf9f16d31291e-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.014 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:01.015 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:01.015 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.015 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:17:01.015 DEBUG:boto:Data: 2024-03-04T16:17:01.015 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.015 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.015 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.015 DEBUG:boto:Params: {} 2024-03-04T16:17:01.015 DEBUG:boto:Token: None 2024-03-04T16:17:01.015 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o 2024-03-04T16:17:01.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d895qPjW/q1aUyDT5OX9dbubdAg= 2024-03-04T16:17:01.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d895qPjW/q1aUyDT5OX9dbubdAg=', 'Content-Length': '0'} 2024-03-04T16:17:01.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003e50ab8d4b1c879-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-03-04T16:17:01.018 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:01.018 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:01.018 DEBUG:boto:Method: GET 2024-03-04T16:17:01.018 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:17:01.018 DEBUG:boto:Data: 2024-03-04T16:17:01.018 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.018 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.018 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.018 DEBUG:boto:Params: {} 2024-03-04T16:17:01.018 DEBUG:boto:Token: None 2024-03-04T16:17:01.019 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o 2024-03-04T16:17:01.019 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2jx4vO99FWcmzYQmUp3p+UB8zfU= 2024-03-04T16:17:01.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2jx4vO99FWcmzYQmUp3p+UB8zfU=', 'Content-Length': '0'} 2024-03-04T16:17:01.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbf6b344330eff5b-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.021 DEBUG:boto:path=/sbnvju-12/.o 2024-03-04T16:17:01.021 DEBUG:boto:auth_path=/sbnvju-12/.o 2024-03-04T16:17:01.021 DEBUG:boto:Method: GET 2024-03-04T16:17:01.021 DEBUG:boto:Path: /sbnvju-12/.o 2024-03-04T16:17:01.021 DEBUG:boto:Data: 2024-03-04T16:17:01.021 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.021 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.021 DEBUG:boto:Params: {} 2024-03-04T16:17:01.021 DEBUG:boto:Token: None 2024-03-04T16:17:01.022 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o 2024-03-04T16:17:01.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2jx4vO99FWcmzYQmUp3p+UB8zfU= 2024-03-04T16:17:01.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2jx4vO99FWcmzYQmUp3p+UB8zfU=', 'Content-Length': '0'} 2024-03-04T16:17:01.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c1f23e2705d950e-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.024 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.024 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.024 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.024 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.024 DEBUG:boto:Method: GET 2024-03-04T16:17:01.024 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:17:01.024 DEBUG:boto:Data: 2024-03-04T16:17:01.025 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.025 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.025 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.025 DEBUG:boto:Params: {} 2024-03-04T16:17:01.025 DEBUG:boto:Token: None 2024-03-04T16:17:01.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f4f5d11299492c-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.028 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.028 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.028 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.028 DEBUG:boto:path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.028 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o 2024-03-04T16:17:01.028 DEBUG:boto:Method: GET 2024-03-04T16:17:01.028 DEBUG:boto:Path: /sbnvju-12/?prefix=.o 2024-03-04T16:17:01.029 DEBUG:boto:Data: 2024-03-04T16:17:01.029 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.029 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.029 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.029 DEBUG:boto:Params: {} 2024-03-04T16:17:01.029 DEBUG:boto:Token: None 2024-03-04T16:17:01.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.029 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a970a5579f2e4cc8-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.031 DEBUG:boto:b'sbnvju-12.o1000false.o2024-03-04T16:13:01.804Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:17:01.032 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.032 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.032 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.032 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.032 DEBUG:boto:Method: GET 2024-03-04T16:17:01.032 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.033 DEBUG:boto:Data: 2024-03-04T16:17:01.033 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.033 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.033 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.033 DEBUG:boto:Params: {} 2024-03-04T16:17:01.033 DEBUG:boto:Token: None 2024-03-04T16:17:01.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.033 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc= 2024-03-04T16:17:01.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc=', 'Content-Length': '0'} 2024-03-04T16:17:01.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e0da5a42bb77cb7-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.035 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.035 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.035 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.036 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.036 DEBUG:boto:Method: GET 2024-03-04T16:17:01.036 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.036 DEBUG:boto:Data: 2024-03-04T16:17:01.036 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.036 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.036 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.036 DEBUG:boto:Params: {} 2024-03-04T16:17:01.036 DEBUG:boto:Token: None 2024-03-04T16:17:01.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.037 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc= 2024-03-04T16:17:01.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc=', 'Content-Length': '0'} 2024-03-04T16:17:01.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ed99d50c66a5ae5-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.039 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.039 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.039 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.039 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.039 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.039 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.039 DEBUG:boto:Data: 2024-03-04T16:17:01.039 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.039 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.039 DEBUG:boto:Params: {} 2024-03-04T16:17:01.039 DEBUG:boto:Token: None 2024-03-04T16:17:01.040 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.040 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iVPE+NOnu4+5OvrIQuRveIOzFwU= 2024-03-04T16:17:01.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iVPE+NOnu4+5OvrIQuRveIOzFwU=', 'Content-Length': '0'} 2024-03-04T16:17:01.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2a4148a99f49a8e-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.042 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.042 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.042 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.042 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.042 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.042 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.042 DEBUG:boto:Data: 2024-03-04T16:17:01.042 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.042 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.042 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.042 DEBUG:boto:Params: {} 2024-03-04T16:17:01.043 DEBUG:boto:Token: None 2024-03-04T16:17:01.043 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.043 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iVPE+NOnu4+5OvrIQuRveIOzFwU= 2024-03-04T16:17:01.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iVPE+NOnu4+5OvrIQuRveIOzFwU=', 'Content-Length': '0'} 2024-03-04T16:17:01.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a44e941f348563af-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:17:01.045 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.045 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.045 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.045 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.045 DEBUG:boto:Method: GET 2024-03-04T16:17:01.045 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.045 DEBUG:boto:Data: 2024-03-04T16:17:01.045 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.045 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.045 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.045 DEBUG:boto:Params: {} 2024-03-04T16:17:01.046 DEBUG:boto:Token: None 2024-03-04T16:17:01.046 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc= 2024-03-04T16:17:01.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc=', 'Content-Length': '0'} 2024-03-04T16:17:01.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f20f6777da718ae-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.048 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.048 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.048 DEBUG:boto:path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.048 DEBUG:boto:auth_path=/sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.048 DEBUG:boto:Method: GET 2024-03-04T16:17:01.048 DEBUG:boto:Path: /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.048 DEBUG:boto:Data: 2024-03-04T16:17:01.048 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.048 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.048 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.049 DEBUG:boto:Params: {} 2024-03-04T16:17:01.049 DEBUG:boto:Token: None 2024-03-04T16:17:01.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o.?versionId=null 2024-03-04T16:17:01.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc= 2024-03-04T16:17:01.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oVbosbF4K7lhO6hvKs7iApxwACc=', 'Content-Length': '0'} 2024-03-04T16:17:01.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c0a320c928fdfa4-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.051 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.051 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.051 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.051 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.051 DEBUG:boto:Method: GET 2024-03-04T16:17:01.051 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.051 DEBUG:boto:Data: 2024-03-04T16:17:01.051 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.052 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.052 DEBUG:boto:Params: {} 2024-03-04T16:17:01.052 DEBUG:boto:Token: None 2024-03-04T16:17:01.052 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.052 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307a177c7d503482-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.054 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.055 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.055 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.055 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.055 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.055 DEBUG:boto:Method: GET 2024-03-04T16:17:01.055 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.055 DEBUG:boto:Data: 2024-03-04T16:17:01.055 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.056 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.056 DEBUG:boto:Params: {} 2024-03-04T16:17:01.056 DEBUG:boto:Token: None 2024-03-04T16:17:01.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.056 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f2af16fbc0b20b-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.059 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.059 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.059 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.059 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.059 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:17:01.059 DEBUG:boto:Data: 2024-03-04T16:17:01.059 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.059 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.059 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.060 DEBUG:boto:Params: {} 2024-03-04T16:17:01.060 DEBUG:boto:Token: None 2024-03-04T16:17:01.060 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o. 2024-03-04T16:17:01.060 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVXKgkT8lgPnihqF6AiNIGS24rg= 2024-03-04T16:17:01.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVXKgkT8lgPnihqF6AiNIGS24rg=', 'Content-Length': '0'} 2024-03-04T16:17:01.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018192fb8dab3d0d8-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.062 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.062 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.062 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.062 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:17:01.062 DEBUG:boto:Data: 2024-03-04T16:17:01.062 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.062 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.062 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.062 DEBUG:boto:Params: {} 2024-03-04T16:17:01.062 DEBUG:boto:Token: None 2024-03-04T16:17:01.063 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o. 2024-03-04T16:17:01.063 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVXKgkT8lgPnihqF6AiNIGS24rg= 2024-03-04T16:17:01.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVXKgkT8lgPnihqF6AiNIGS24rg=', 'Content-Length': '0'} 2024-03-04T16:17:01.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093e809057db2cf58-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-03-04T16:17:01.065 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.065 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.065 DEBUG:boto:Method: GET 2024-03-04T16:17:01.065 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:17:01.065 DEBUG:boto:Data: 2024-03-04T16:17:01.065 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.065 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.065 DEBUG:boto:Params: {} 2024-03-04T16:17:01.065 DEBUG:boto:Token: None 2024-03-04T16:17:01.066 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o. 2024-03-04T16:17:01.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N6NC777rztDZ9eBubSOCpoU3sMQ= 2024-03-04T16:17:01.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N6NC777rztDZ9eBubSOCpoU3sMQ=', 'Content-Length': '0'} 2024-03-04T16:17:01.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001aa12136a6931bdd-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.068 DEBUG:boto:path=/sbnvju-12/.o. 2024-03-04T16:17:01.068 DEBUG:boto:auth_path=/sbnvju-12/.o. 2024-03-04T16:17:01.068 DEBUG:boto:Method: GET 2024-03-04T16:17:01.068 DEBUG:boto:Path: /sbnvju-12/.o. 2024-03-04T16:17:01.069 DEBUG:boto:Data: 2024-03-04T16:17:01.069 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.069 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.069 DEBUG:boto:Params: {} 2024-03-04T16:17:01.069 DEBUG:boto:Token: None 2024-03-04T16:17:01.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/.o. 2024-03-04T16:17:01.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N6NC777rztDZ9eBubSOCpoU3sMQ= 2024-03-04T16:17:01.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N6NC777rztDZ9eBubSOCpoU3sMQ=', 'Content-Length': '0'} 2024-03-04T16:17:01.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acde31882b95a7d1-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.112 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.112 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.112 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.113 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.113 DEBUG:boto:Method: GET 2024-03-04T16:17:01.113 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.113 DEBUG:boto:Data: 2024-03-04T16:17:01.113 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.114 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.114 DEBUG:boto:Params: {} 2024-03-04T16:17:01.114 DEBUG:boto:Token: None 2024-03-04T16:17:01.114 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.114 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd51cd50d5b93feb-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.117 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.118 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.118 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.118 DEBUG:boto:path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.118 DEBUG:boto:auth_path=/sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.118 DEBUG:boto:Method: GET 2024-03-04T16:17:01.118 DEBUG:boto:Path: /sbnvju-12/?prefix=.o. 2024-03-04T16:17:01.119 DEBUG:boto:Data: 2024-03-04T16:17:01.119 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.119 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.119 DEBUG:boto:Params: {} 2024-03-04T16:17:01.119 DEBUG:boto:Token: None 2024-03-04T16:17:01.119 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef76b4aef6fc976-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.122 DEBUG:boto:b'sbnvju-12.o.1000false.o.2024-03-04T16:13:01.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:17:01.123 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.123 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.123 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.123 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.124 DEBUG:boto:Method: GET 2024-03-04T16:17:01.124 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.124 DEBUG:boto:Data: 2024-03-04T16:17:01.124 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.124 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.125 DEBUG:boto:Params: {} 2024-03-04T16:17:01.125 DEBUG:boto:Token: None 2024-03-04T16:17:01.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs= 2024-03-04T16:17:01.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs=', 'Content-Length': '0'} 2024-03-04T16:17:01.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e92056a9efb5cc67-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.168 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.169 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.169 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.169 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.169 DEBUG:boto:Method: GET 2024-03-04T16:17:01.169 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.170 DEBUG:boto:Data: 2024-03-04T16:17:01.170 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.170 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.170 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.170 DEBUG:boto:Params: {} 2024-03-04T16:17:01.171 DEBUG:boto:Token: None 2024-03-04T16:17:01.171 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.171 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs= 2024-03-04T16:17:01.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs=', 'Content-Length': '0'} 2024-03-04T16:17:01.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006631b13d78116558-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.174 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.174 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.174 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.174 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.174 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.174 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.174 DEBUG:boto:Data: 2024-03-04T16:17:01.174 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.174 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.175 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.175 DEBUG:boto:Params: {} 2024-03-04T16:17:01.175 DEBUG:boto:Token: None 2024-03-04T16:17:01.175 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.175 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NMMQ2w+edUASgla+hWtGoA1VdoE= 2024-03-04T16:17:01.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NMMQ2w+edUASgla+hWtGoA1VdoE=', 'Content-Length': '0'} 2024-03-04T16:17:01.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9a395b55ee8f92a-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.177 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.177 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.177 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.177 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.177 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.177 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.177 DEBUG:boto:Data: 2024-03-04T16:17:01.177 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.178 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.178 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.178 DEBUG:boto:Params: {} 2024-03-04T16:17:01.178 DEBUG:boto:Token: None 2024-03-04T16:17:01.178 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NMMQ2w+edUASgla+hWtGoA1VdoE= 2024-03-04T16:17:01.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NMMQ2w+edUASgla+hWtGoA1VdoE=', 'Content-Length': '0'} 2024-03-04T16:17:01.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077352504a75004f7-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:17:01.181 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.181 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.181 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.181 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.181 DEBUG:boto:Method: GET 2024-03-04T16:17:01.181 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.181 DEBUG:boto:Data: 2024-03-04T16:17:01.181 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.181 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.182 DEBUG:boto:Params: {} 2024-03-04T16:17:01.182 DEBUG:boto:Token: None 2024-03-04T16:17:01.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs= 2024-03-04T16:17:01.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs=', 'Content-Length': '0'} 2024-03-04T16:17:01.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c05d3d1251373301-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.184 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.184 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.184 DEBUG:boto:path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.184 DEBUG:boto:auth_path=/sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.185 DEBUG:boto:Method: GET 2024-03-04T16:17:01.185 DEBUG:boto:Path: /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.185 DEBUG:boto:Data: 2024-03-04T16:17:01.185 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.185 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.185 DEBUG:boto:Params: {} 2024-03-04T16:17:01.185 DEBUG:boto:Token: None 2024-03-04T16:17:01.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A?versionId=null 2024-03-04T16:17:01.185 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs= 2024-03-04T16:17:01.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d9WreoXE9/deTib49I6idz/FuMs=', 'Content-Length': '0'} 2024-03-04T16:17:01.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005d81565c369677b-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.188 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.188 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.188 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.188 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.188 DEBUG:boto:Method: GET 2024-03-04T16:17:01.188 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.188 DEBUG:boto:Data: 2024-03-04T16:17:01.188 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.188 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.189 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.189 DEBUG:boto:Params: {} 2024-03-04T16:17:01.189 DEBUG:boto:Token: None 2024-03-04T16:17:01.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0fc04bbdf48f932-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.232 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.233 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.233 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.233 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.233 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.233 DEBUG:boto:Method: GET 2024-03-04T16:17:01.233 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.233 DEBUG:boto:Data: 2024-03-04T16:17:01.233 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.234 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.234 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.234 DEBUG:boto:Params: {} 2024-03-04T16:17:01.234 DEBUG:boto:Token: None 2024-03-04T16:17:01.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1548c23666146c-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.237 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.237 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.238 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.238 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.238 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:01.238 DEBUG:boto:Data: 2024-03-04T16:17:01.238 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.238 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.238 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.238 DEBUG:boto:Params: {} 2024-03-04T16:17:01.238 DEBUG:boto:Token: None 2024-03-04T16:17:01.238 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A 2024-03-04T16:17:01.239 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BGPBDep/rVu1bof8Bv3sekjD0TE= 2024-03-04T16:17:01.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BGPBDep/rVu1bof8Bv3sekjD0TE=', 'Content-Length': '0'} 2024-03-04T16:17:01.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2dd34b3c5ecd139-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.241 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.241 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.241 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.241 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:01.241 DEBUG:boto:Data: 2024-03-04T16:17:01.241 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.241 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.241 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.241 DEBUG:boto:Params: {} 2024-03-04T16:17:01.242 DEBUG:boto:Token: None 2024-03-04T16:17:01.242 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A 2024-03-04T16:17:01.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BGPBDep/rVu1bof8Bv3sekjD0TE= 2024-03-04T16:17:01.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BGPBDep/rVu1bof8Bv3sekjD0TE=', 'Content-Length': '0'} 2024-03-04T16:17:01.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004a7ebd8c63a3c81-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-03-04T16:17:01.244 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.244 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.244 DEBUG:boto:Method: GET 2024-03-04T16:17:01.244 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:01.244 DEBUG:boto:Data: 2024-03-04T16:17:01.245 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.245 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.245 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.245 DEBUG:boto:Params: {} 2024-03-04T16:17:01.245 DEBUG:boto:Token: None 2024-03-04T16:17:01.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A 2024-03-04T16:17:01.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H/OtFLNwLE4FATsQmiWjINYS4co= 2024-03-04T16:17:01.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H/OtFLNwLE4FATsQmiWjINYS4co=', 'Content-Length': '0'} 2024-03-04T16:17:01.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e50474e87b2081c1-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.247 DEBUG:boto:path=/sbnvju-12/%3A 2024-03-04T16:17:01.247 DEBUG:boto:auth_path=/sbnvju-12/%3A 2024-03-04T16:17:01.247 DEBUG:boto:Method: GET 2024-03-04T16:17:01.247 DEBUG:boto:Path: /sbnvju-12/%3A 2024-03-04T16:17:01.247 DEBUG:boto:Data: 2024-03-04T16:17:01.247 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.248 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.248 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.248 DEBUG:boto:Params: {} 2024-03-04T16:17:01.248 DEBUG:boto:Token: None 2024-03-04T16:17:01.248 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/%3A 2024-03-04T16:17:01.248 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H/OtFLNwLE4FATsQmiWjINYS4co= 2024-03-04T16:17:01.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H/OtFLNwLE4FATsQmiWjINYS4co=', 'Content-Length': '0'} 2024-03-04T16:17:01.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebc3afc64986e533-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.250 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.250 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.251 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.251 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.251 DEBUG:boto:Method: GET 2024-03-04T16:17:01.251 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.251 DEBUG:boto:Data: 2024-03-04T16:17:01.251 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.251 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.251 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.251 DEBUG:boto:Params: {} 2024-03-04T16:17:01.251 DEBUG:boto:Token: None 2024-03-04T16:17:01.251 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.252 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3be1a821ca3def-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.254 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.255 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.255 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.255 DEBUG:boto:path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.255 DEBUG:boto:auth_path=/sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.255 DEBUG:boto:Method: GET 2024-03-04T16:17:01.255 DEBUG:boto:Path: /sbnvju-12/?prefix=%3A 2024-03-04T16:17:01.255 DEBUG:boto:Data: 2024-03-04T16:17:01.255 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.255 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.255 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.255 DEBUG:boto:Params: {} 2024-03-04T16:17:01.255 DEBUG:boto:Token: None 2024-03-04T16:17:01.256 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.256 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005218658f3706e323-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.258 DEBUG:boto:b'sbnvju-12:1000false:2024-03-04T16:13:01.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:01.259 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.259 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.259 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.259 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.259 DEBUG:boto:Method: GET 2024-03-04T16:17:01.259 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.259 DEBUG:boto:Data: 2024-03-04T16:17:01.260 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.260 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.260 DEBUG:boto:Params: {} 2024-03-04T16:17:01.260 DEBUG:boto:Token: None 2024-03-04T16:17:01.260 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.260 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA= 2024-03-04T16:17:01.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA=', 'Content-Length': '0'} 2024-03-04T16:17:01.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016084d261de1173b-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.263 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.263 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.263 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.263 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.263 DEBUG:boto:Method: GET 2024-03-04T16:17:01.263 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.263 DEBUG:boto:Data: 2024-03-04T16:17:01.263 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.263 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.263 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.263 DEBUG:boto:Params: {} 2024-03-04T16:17:01.264 DEBUG:boto:Token: None 2024-03-04T16:17:01.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA= 2024-03-04T16:17:01.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA=', 'Content-Length': '0'} 2024-03-04T16:17:01.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000495bc3672985829d-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.266 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.266 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.266 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.266 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.266 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.266 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.266 DEBUG:boto:Data: 2024-03-04T16:17:01.266 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.266 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.266 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.266 DEBUG:boto:Params: {} 2024-03-04T16:17:01.267 DEBUG:boto:Token: None 2024-03-04T16:17:01.267 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PW00vzZGkzz0+hQoIAoftnE90/I= 2024-03-04T16:17:01.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PW00vzZGkzz0+hQoIAoftnE90/I=', 'Content-Length': '0'} 2024-03-04T16:17:01.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb90beff49a03ee-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.269 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.269 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.269 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.269 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.270 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.270 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.270 DEBUG:boto:Data: 2024-03-04T16:17:01.270 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.270 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.270 DEBUG:boto:Params: {} 2024-03-04T16:17:01.270 DEBUG:boto:Token: None 2024-03-04T16:17:01.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PW00vzZGkzz0+hQoIAoftnE90/I= 2024-03-04T16:17:01.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PW00vzZGkzz0+hQoIAoftnE90/I=', 'Content-Length': '0'} 2024-03-04T16:17:01.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c7700c3bfcaf3ea-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:01.273 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.273 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.273 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.273 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.273 DEBUG:boto:Method: GET 2024-03-04T16:17:01.273 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.273 DEBUG:boto:Data: 2024-03-04T16:17:01.273 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.274 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.274 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.274 DEBUG:boto:Params: {} 2024-03-04T16:17:01.274 DEBUG:boto:Token: None 2024-03-04T16:17:01.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA= 2024-03-04T16:17:01.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA=', 'Content-Length': '0'} 2024-03-04T16:17:01.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afad6f698827f262-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.320 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.320 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.321 DEBUG:boto:path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.321 DEBUG:boto:auth_path=/sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.321 DEBUG:boto:Method: GET 2024-03-04T16:17:01.321 DEBUG:boto:Path: /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.321 DEBUG:boto:Data: 2024-03-04T16:17:01.322 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.322 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.322 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.322 DEBUG:boto:Params: {} 2024-03-04T16:17:01.322 DEBUG:boto:Token: None 2024-03-04T16:17:01.323 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj?versionId=null 2024-03-04T16:17:01.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA= 2024-03-04T16:17:01.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9HeelI9Sq1yFCOKO+KETfHgJryA=', 'Content-Length': '0'} 2024-03-04T16:17:01.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000250e5bde8f2a131f-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.325 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.326 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.326 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.326 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.326 DEBUG:boto:Method: GET 2024-03-04T16:17:01.326 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.326 DEBUG:boto:Data: 2024-03-04T16:17:01.327 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.327 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.327 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.327 DEBUG:boto:Params: {} 2024-03-04T16:17:01.327 DEBUG:boto:Token: None 2024-03-04T16:17:01.328 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.328 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d979b97a27221f1-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.331 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.331 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.331 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.332 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.332 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.332 DEBUG:boto:Method: GET 2024-03-04T16:17:01.332 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.332 DEBUG:boto:Data: 2024-03-04T16:17:01.333 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.333 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.333 DEBUG:boto:Params: {} 2024-03-04T16:17:01.333 DEBUG:boto:Token: None 2024-03-04T16:17:01.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.334 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f103d582a1821b9-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.380 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.381 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.381 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.381 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.381 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:01.381 DEBUG:boto:Data: 2024-03-04T16:17:01.381 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.381 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.381 DEBUG:boto:Params: {} 2024-03-04T16:17:01.381 DEBUG:boto:Token: None 2024-03-04T16:17:01.382 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj 2024-03-04T16:17:01.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+/T6+GD3CFEs9clg5bcdTZXjd/k= 2024-03-04T16:17:01.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+/T6+GD3CFEs9clg5bcdTZXjd/k=', 'Content-Length': '0'} 2024-03-04T16:17:01.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e59bb6510d52a90-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.384 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.384 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.384 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.384 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:01.384 DEBUG:boto:Data: 2024-03-04T16:17:01.384 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.385 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.385 DEBUG:boto:Params: {} 2024-03-04T16:17:01.385 DEBUG:boto:Token: None 2024-03-04T16:17:01.385 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj 2024-03-04T16:17:01.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+/T6+GD3CFEs9clg5bcdTZXjd/k= 2024-03-04T16:17:01.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+/T6+GD3CFEs9clg5bcdTZXjd/k=', 'Content-Length': '0'} 2024-03-04T16:17:01.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000275c6da2acd28424-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-03-04T16:17:01.387 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.387 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.387 DEBUG:boto:Method: GET 2024-03-04T16:17:01.387 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:01.387 DEBUG:boto:Data: 2024-03-04T16:17:01.387 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.388 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.388 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.388 DEBUG:boto:Params: {} 2024-03-04T16:17:01.388 DEBUG:boto:Token: None 2024-03-04T16:17:01.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj 2024-03-04T16:17:01.388 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9qU0Urf0PQ1WJIEuM63jHwffu2o= 2024-03-04T16:17:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9qU0Urf0PQ1WJIEuM63jHwffu2o=', 'Content-Length': '0'} 2024-03-04T16:17:01.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d432a5c8c17d3140-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.390 DEBUG:boto:path=/sbnvju-12/_myobj 2024-03-04T16:17:01.390 DEBUG:boto:auth_path=/sbnvju-12/_myobj 2024-03-04T16:17:01.391 DEBUG:boto:Method: GET 2024-03-04T16:17:01.391 DEBUG:boto:Path: /sbnvju-12/_myobj 2024-03-04T16:17:01.391 DEBUG:boto:Data: 2024-03-04T16:17:01.391 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.391 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.391 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.391 DEBUG:boto:Params: {} 2024-03-04T16:17:01.391 DEBUG:boto:Token: None 2024-03-04T16:17:01.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/_myobj 2024-03-04T16:17:01.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9qU0Urf0PQ1WJIEuM63jHwffu2o= 2024-03-04T16:17:01.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9qU0Urf0PQ1WJIEuM63jHwffu2o=', 'Content-Length': '0'} 2024-03-04T16:17:01.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080471d6ef8a05c8f-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.393 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.393 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.394 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.394 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.394 DEBUG:boto:Method: GET 2024-03-04T16:17:01.394 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.394 DEBUG:boto:Data: 2024-03-04T16:17:01.394 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.394 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.394 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.394 DEBUG:boto:Params: {} 2024-03-04T16:17:01.395 DEBUG:boto:Token: None 2024-03-04T16:17:01.395 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aee8fc35c87fce1-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.397 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.398 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.398 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.398 DEBUG:boto:path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.398 DEBUG:boto:auth_path=/sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.398 DEBUG:boto:Method: GET 2024-03-04T16:17:01.398 DEBUG:boto:Path: /sbnvju-12/?prefix=_myobj 2024-03-04T16:17:01.398 DEBUG:boto:Data: 2024-03-04T16:17:01.399 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.399 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.399 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.399 DEBUG:boto:Params: {} 2024-03-04T16:17:01.399 DEBUG:boto:Token: None 2024-03-04T16:17:01.399 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.399 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ea5310e6401f55-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.402 DEBUG:boto:b'sbnvju-12_myobj1000false_myobj2024-03-04T16:13:01.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:01.403 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.403 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.403 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.404 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.404 DEBUG:boto:Method: GET 2024-03-04T16:17:01.404 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.404 DEBUG:boto:Data: 2024-03-04T16:17:01.404 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.405 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.405 DEBUG:boto:Params: {} 2024-03-04T16:17:01.405 DEBUG:boto:Token: None 2024-03-04T16:17:01.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo= 2024-03-04T16:17:01.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo=', 'Content-Length': '0'} 2024-03-04T16:17:01.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc25db3c422529c3-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.408 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.408 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.409 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.409 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.409 DEBUG:boto:Method: GET 2024-03-04T16:17:01.409 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.409 DEBUG:boto:Data: 2024-03-04T16:17:01.409 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.409 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.409 DEBUG:boto:Params: {} 2024-03-04T16:17:01.409 DEBUG:boto:Token: None 2024-03-04T16:17:01.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo= 2024-03-04T16:17:01.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo=', 'Content-Length': '0'} 2024-03-04T16:17:01.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74eb09fc123b777-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.412 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.412 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.412 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.412 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.412 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.412 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.412 DEBUG:boto:Data: 2024-03-04T16:17:01.412 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.412 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.413 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.413 DEBUG:boto:Params: {} 2024-03-04T16:17:01.413 DEBUG:boto:Token: None 2024-03-04T16:17:01.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VaVTcCKA7sXrMaNJMynWzWnakd4= 2024-03-04T16:17:01.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VaVTcCKA7sXrMaNJMynWzWnakd4=', 'Content-Length': '0'} 2024-03-04T16:17:01.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000013875ccbe99f583-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.415 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.416 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.416 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.416 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.416 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.416 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.417 DEBUG:boto:Data: 2024-03-04T16:17:01.417 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.417 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.417 DEBUG:boto:Params: {} 2024-03-04T16:17:01.417 DEBUG:boto:Token: None 2024-03-04T16:17:01.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VaVTcCKA7sXrMaNJMynWzWnakd4= 2024-03-04T16:17:01.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VaVTcCKA7sXrMaNJMynWzWnakd4=', 'Content-Length': '0'} 2024-03-04T16:17:01.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e334cda93284a90a-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:01.420 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.420 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.420 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.420 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.420 DEBUG:boto:Method: GET 2024-03-04T16:17:01.420 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.420 DEBUG:boto:Data: 2024-03-04T16:17:01.420 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.421 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.421 DEBUG:boto:Params: {} 2024-03-04T16:17:01.421 DEBUG:boto:Token: None 2024-03-04T16:17:01.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo= 2024-03-04T16:17:01.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo=', 'Content-Length': '0'} 2024-03-04T16:17:01.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050c3a3997546304f-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.423 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.423 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.423 DEBUG:boto:path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.423 DEBUG:boto:auth_path=/sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.423 DEBUG:boto:Method: GET 2024-03-04T16:17:01.423 DEBUG:boto:Path: /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.423 DEBUG:boto:Data: 2024-03-04T16:17:01.424 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.424 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.424 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.424 DEBUG:boto:Params: {} 2024-03-04T16:17:01.424 DEBUG:boto:Token: None 2024-03-04T16:17:01.424 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj?versionId=null 2024-03-04T16:17:01.424 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo= 2024-03-04T16:17:01.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h8fRLf91L10WBg7kf2FP6KlPWRo=', 'Content-Length': '0'} 2024-03-04T16:17:01.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a9512ecf7c71c6c-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.426 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.427 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.427 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.427 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.427 DEBUG:boto:Method: GET 2024-03-04T16:17:01.427 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.427 DEBUG:boto:Data: 2024-03-04T16:17:01.427 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.427 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.427 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.427 DEBUG:boto:Params: {} 2024-03-04T16:17:01.427 DEBUG:boto:Token: None 2024-03-04T16:17:01.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.428 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638d6510a19856a0-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.430 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.431 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.431 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.431 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.431 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.431 DEBUG:boto:Method: GET 2024-03-04T16:17:01.431 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.431 DEBUG:boto:Data: 2024-03-04T16:17:01.431 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.431 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.431 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.431 DEBUG:boto:Params: {} 2024-03-04T16:17:01.432 DEBUG:boto:Token: None 2024-03-04T16:17:01.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042bc33a798d6eff7-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.434 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.434 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.435 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.435 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.435 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:01.435 DEBUG:boto:Data: 2024-03-04T16:17:01.435 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.435 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.435 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.435 DEBUG:boto:Params: {} 2024-03-04T16:17:01.435 DEBUG:boto:Token: None 2024-03-04T16:17:01.435 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj 2024-03-04T16:17:01.436 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xeXdYCddU3VCAmvc8iY3ofGqWtI= 2024-03-04T16:17:01.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xeXdYCddU3VCAmvc8iY3ofGqWtI=', 'Content-Length': '0'} 2024-03-04T16:17:01.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2ee29c5983ae850-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.438 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.438 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.438 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.438 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:01.438 DEBUG:boto:Data: 2024-03-04T16:17:01.438 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.438 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.438 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.438 DEBUG:boto:Params: {} 2024-03-04T16:17:01.439 DEBUG:boto:Token: None 2024-03-04T16:17:01.439 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj 2024-03-04T16:17:01.439 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xeXdYCddU3VCAmvc8iY3ofGqWtI= 2024-03-04T16:17:01.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xeXdYCddU3VCAmvc8iY3ofGqWtI=', 'Content-Length': '0'} 2024-03-04T16:17:01.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005664b7f0033e5a96-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:01.441 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.441 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.441 DEBUG:boto:Method: GET 2024-03-04T16:17:01.441 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:01.441 DEBUG:boto:Data: 2024-03-04T16:17:01.441 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.442 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.442 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.442 DEBUG:boto:Params: {} 2024-03-04T16:17:01.442 DEBUG:boto:Token: None 2024-03-04T16:17:01.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj 2024-03-04T16:17:01.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:53AVLSDMH+BLoC4ln287yEWFvt4= 2024-03-04T16:17:01.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:53AVLSDMH+BLoC4ln287yEWFvt4=', 'Content-Length': '0'} 2024-03-04T16:17:01.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-12:fc294220-5038-4f25-aeee-cac2c5838729.4341.21'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f8ec2256c9e13d4-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.444 DEBUG:boto:path=/sbnvju-12/myobj 2024-03-04T16:17:01.444 DEBUG:boto:auth_path=/sbnvju-12/myobj 2024-03-04T16:17:01.444 DEBUG:boto:Method: GET 2024-03-04T16:17:01.444 DEBUG:boto:Path: /sbnvju-12/myobj 2024-03-04T16:17:01.445 DEBUG:boto:Data: 2024-03-04T16:17:01.445 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.445 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.445 DEBUG:boto:Params: {} 2024-03-04T16:17:01.445 DEBUG:boto:Token: None 2024-03-04T16:17:01.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/myobj 2024-03-04T16:17:01.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:53AVLSDMH+BLoC4ln287yEWFvt4= 2024-03-04T16:17:01.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:53AVLSDMH+BLoC4ln287yEWFvt4=', 'Content-Length': '0'} 2024-03-04T16:17:01.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:13:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000089171cce74716b2-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.488 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.488 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.488 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.489 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.489 DEBUG:boto:Method: GET 2024-03-04T16:17:01.489 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.489 DEBUG:boto:Data: 2024-03-04T16:17:01.489 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.489 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.489 DEBUG:boto:Params: {} 2024-03-04T16:17:01.489 DEBUG:boto:Token: None 2024-03-04T16:17:01.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.490 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbeeb0e51787b2dd-0065e5f3fd-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.492 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.493 DEBUG:boto:path=/sbnvju-12/ 2024-03-04T16:17:01.493 DEBUG:boto:auth_path=/sbnvju-12/ 2024-03-04T16:17:01.493 DEBUG:boto:path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.493 DEBUG:boto:auth_path=/sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.493 DEBUG:boto:Method: GET 2024-03-04T16:17:01.493 DEBUG:boto:Path: /sbnvju-12/?prefix=myobj 2024-03-04T16:17:01.493 DEBUG:boto:Data: 2024-03-04T16:17:01.493 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.493 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.493 DEBUG:boto:Params: {} 2024-03-04T16:17:01.494 DEBUG:boto:Token: None 2024-03-04T16:17:01.494 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-12/ 2024-03-04T16:17:01.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w= 2024-03-04T16:17:01.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2RAyt0mbmtogH8ft/GuU3JfD89w=', 'Content-Length': '0'} 2024-03-04T16:17:01.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b4c237dda58fa8-0065e5f3fd-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.496 DEBUG:boto:b'sbnvju-12myobj1000falsemyobj2024-03-04T16:13:01.592Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:01.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-12 zones={test-zone2, test-zone3} 2024-03-04T16:17:01.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-03-04T16:17:01.499 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-13 2024-03-04T16:17:01.499 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:01.499 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:01.499 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.499 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:01.500 DEBUG:boto:Data: 2024-03-04T16:17:01.500 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:01.500 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.500 DEBUG:boto:Params: {} 2024-03-04T16:17:01.500 DEBUG:boto:Token: None 2024-03-04T16:17:01.500 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-13/ 2024-03-04T16:17:01.500 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nx+Dt10BdCtoMsdOKoYiczR7iF8= 2024-03-04T16:17:01.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nx+Dt10BdCtoMsdOKoYiczR7iF8=', 'Content-Length': '0'} 2024-03-04T16:17:01.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094b1e654044bf458-0065e5f3fd-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.517 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-14 2024-03-04T16:17:01.517 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:17:01.517 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:17:01.517 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.517 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:17:01.518 DEBUG:boto:Data: 2024-03-04T16:17:01.518 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.518 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.518 DEBUG:boto:Params: {} 2024-03-04T16:17:01.518 DEBUG:boto:Token: None 2024-03-04T16:17:01.518 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-14/ 2024-03-04T16:17:01.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V21LtipgzCMszIsBS2+O3NE7BSI= 2024-03-04T16:17:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V21LtipgzCMszIsBS2+O3NE7BSI=', 'Content-Length': '0'} 2024-03-04T16:17:01.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4adbbe8816300d0-0065e5f3fd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.549 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-15 2024-03-04T16:17:01.549 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:17:01.549 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:17:01.549 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.549 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:17:01.550 DEBUG:boto:Data: 2024-03-04T16:17:01.550 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.550 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.550 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.550 DEBUG:boto:Params: {} 2024-03-04T16:17:01.550 DEBUG:boto:Token: None 2024-03-04T16:17:01.550 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-15/ 2024-03-04T16:17:01.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nMUNKpUGv5PkDrPxkjl/uBfvMeA= 2024-03-04T16:17:01.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nMUNKpUGv5PkDrPxkjl/uBfvMeA=', 'Content-Length': '0'} 2024-03-04T16:17:01.575 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da99707eb4eef26d-0065e5f3fd-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.576 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:01.576 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:01.576 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.576 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:01.576 DEBUG:boto:Data: 2024-03-04T16:17:01.576 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.576 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:01.576 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.576 DEBUG:boto:Params: {} 2024-03-04T16:17:01.576 DEBUG:boto:Token: None 2024-03-04T16:17:01.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-13/ 2024-03-04T16:17:01.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cBfLI/qqEPHL84WMdy1Tq16H7dM= 2024-03-04T16:17:01.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cBfLI/qqEPHL84WMdy1Tq16H7dM=', 'Content-Length': '0'} 2024-03-04T16:17:01.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ca2d9d15c942e7-0065e5f3fd-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.579 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:01.579 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:01.580 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.580 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:01.580 DEBUG:boto:Data: 2024-03-04T16:17:01.580 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:17:01.580 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:01.580 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.580 DEBUG:boto:Params: {} 2024-03-04T16:17:01.580 DEBUG:boto:Token: None 2024-03-04T16:17:01.580 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-13/myobj 2024-03-04T16:17:01.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MtlUuFk5JaON3YnvYLLyJDbdxhc= 2024-03-04T16:17:01.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MtlUuFk5JaON3YnvYLLyJDbdxhc='} 2024-03-04T16:17:01.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068ea4b71b856972a-0065e5f3fd-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.628 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:17:01.629 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:17:01.629 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.629 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:17:01.629 DEBUG:boto:Data: 2024-03-04T16:17:01.629 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.629 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.629 DEBUG:boto:Params: {} 2024-03-04T16:17:01.629 DEBUG:boto:Token: None 2024-03-04T16:17:01.630 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-14/ 2024-03-04T16:17:01.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e6uR4DQBHP1GK9TLPaIlD4cV5U8= 2024-03-04T16:17:01.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e6uR4DQBHP1GK9TLPaIlD4cV5U8=', 'Content-Length': '0'} 2024-03-04T16:17:01.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001884776f5f68ef87-0065e5f3fd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.632 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:17:01.632 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:17:01.632 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.633 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:17:01.633 DEBUG:boto:Data: 2024-03-04T16:17:01.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:17:01.633 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:01.633 DEBUG:boto:Port: 8000 2024-03-04T16:17:01.633 DEBUG:boto:Params: {} 2024-03-04T16:17:01.633 DEBUG:boto:Token: None 2024-03-04T16:17:01.633 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-14/myobj 2024-03-04T16:17:01.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e0J0Tx76KxC1LCb5aRnYNNAZp0U= 2024-03-04T16:17:01.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e0J0Tx76KxC1LCb5aRnYNNAZp0U='} 2024-03-04T16:17:01.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007acc3dc002a4dfb3-0065e5f3fd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.640 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:17:01.640 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:17:01.640 DEBUG:boto:Method: HEAD 2024-03-04T16:17:01.640 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:17:01.640 DEBUG:boto:Data: 2024-03-04T16:17:01.641 DEBUG:boto:Headers: {} 2024-03-04T16:17:01.641 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.641 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.641 DEBUG:boto:Params: {} 2024-03-04T16:17:01.641 DEBUG:boto:Token: None 2024-03-04T16:17:01.641 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-15/ 2024-03-04T16:17:01.641 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9XxsgiyKjTWfNwz8EUp3RL24bS8= 2024-03-04T16:17:01.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9XxsgiyKjTWfNwz8EUp3RL24bS8=', 'Content-Length': '0'} 2024-03-04T16:17:01.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000633f72f4282bc493-0065e5f3fd-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.644 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:17:01.644 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:17:01.644 DEBUG:boto:Method: PUT 2024-03-04T16:17:01.644 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:17:01.644 DEBUG:boto:Data: 2024-03-04T16:17:01.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:17:01.644 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:01.644 DEBUG:boto:Port: 8001 2024-03-04T16:17:01.644 DEBUG:boto:Params: {} 2024-03-04T16:17:01.645 DEBUG:boto:Token: None 2024-03-04T16:17:01.645 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:17:01 GMT /sbnvju-15/myobj 2024-03-04T16:17:01.645 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OQ0DpUVP+kFTCpxTivAn6E8pkII= 2024-03-04T16:17:01.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:17:01 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OQ0DpUVP+kFTCpxTivAn6E8pkII='} 2024-03-04T16:17:01.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b74c44e7035aa79-0065e5f3fd-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:01.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.651 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:17:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:01.744 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:17:01.763 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709569021.566562_72.1","last_update":"2024-03-04T16:17:01.566562Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:17:01.764 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709569021.566562_72.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:17:01.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:17:01.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.765 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:17:01.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:01.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:01.855 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:01.855+0000 7fc8e6933800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:17:01.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:01.856+0000 7fc8e6933800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:17:01.875 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569021.566562_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:17:01.566562Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:17:01Z"} 2024-03-04T16:17:01.876 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569021.566562_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:17:01.566562Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:17:01Z"} 2024-03-04T16:17:01.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569021.566562_72.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:17:01.566562Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:17:01.876 DEBUG:rgw_multi.tests:log_status={0: '1_1709569021.566562_72.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:17:01.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569021.566562_72.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:17:01.876 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:17:01.876 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:17:01.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:17:01.877 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:17:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:01.972+0000 7f87b4438800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:17:01.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:01.973+0000 7f87b4438800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:17:01.991 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569021.566562_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:17:01.566562Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:17:01Z"} 2024-03-04T16:17:01.992 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569021.566562_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:17:01.566562Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:17:01Z"} 2024-03-04T16:17:01.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569021.566562_72.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:17:01.566562Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:17:01.992 DEBUG:rgw_multi.tests:log_status={0: '1_1709569021.566562_72.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:17:01.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569021.566562_72.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:17:01.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:17:01.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:01.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:01.993 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:17:02.046 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:02.046 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:02.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:02.095+0000 7f27cca24800 1 waiting to reach incremental sync.. 2024-03-04T16:17:07.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:07.096+0000 7f27cca24800 1 waiting to reach incremental sync.. 2024-03-04T16:17:12.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:12.096+0000 7f27cca24800 1 waiting to reach incremental sync.. 2024-03-04T16:17:17.098 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:17.099+0000 7f27cca24800 1 bucket sync caught up with source: 2024-03-04T16:17:17.098 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:17:17.099 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:17:17.099 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:17:17.099+0000 7f27cca24800 0 bucket checkpoint complete 2024-03-04T16:17:17.121 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-13 zones={test-zone2, test-zone1} 2024-03-04T16:17:17.121 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.121 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.121 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.121 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:17.121 DEBUG:boto:Data: 2024-03-04T16:17:17.121 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.122 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.122 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.122 DEBUG:boto:Params: {} 2024-03-04T16:17:17.122 DEBUG:boto:Token: None 2024-03-04T16:17:17.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DndNFp42xGxDk/Wuvc8dMf2lIJI= 2024-03-04T16:17:17.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DndNFp42xGxDk/Wuvc8dMf2lIJI=', 'Content-Length': '0'} 2024-03-04T16:17:17.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a757e75a5c9f51-0065e5f40d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.125 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.125 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.125 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.126 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:17.126 DEBUG:boto:Data: 2024-03-04T16:17:17.126 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.126 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.126 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.126 DEBUG:boto:Params: {} 2024-03-04T16:17:17.126 DEBUG:boto:Token: None 2024-03-04T16:17:17.126 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DndNFp42xGxDk/Wuvc8dMf2lIJI= 2024-03-04T16:17:17.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DndNFp42xGxDk/Wuvc8dMf2lIJI=', 'Content-Length': '0'} 2024-03-04T16:17:17.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eba0ba3d5be13a3-0065e5f40d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.129 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:17:17.129 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.129 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.130 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.130 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.130 DEBUG:boto:Method: GET 2024-03-04T16:17:17.130 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.130 DEBUG:boto:Data: 2024-03-04T16:17:17.130 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.130 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.130 DEBUG:boto:Params: {} 2024-03-04T16:17:17.130 DEBUG:boto:Token: None 2024-03-04T16:17:17.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/?versions 2024-03-04T16:17:17.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc= 2024-03-04T16:17:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc=', 'Content-Length': '0'} 2024-03-04T16:17:17.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000463b7c753de32774-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.133 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.133 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:17.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:17:17.134 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.134 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.134 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.134 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.134 DEBUG:boto:Method: GET 2024-03-04T16:17:17.134 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.134 DEBUG:boto:Data: 2024-03-04T16:17:17.134 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.134 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.134 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.134 DEBUG:boto:Params: {} 2024-03-04T16:17:17.134 DEBUG:boto:Token: None 2024-03-04T16:17:17.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/?versions 2024-03-04T16:17:17.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc= 2024-03-04T16:17:17.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc=', 'Content-Length': '0'} 2024-03-04T16:17:17.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bdaf20f9764054-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.137 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.138 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:17.138 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.138 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.138 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.138 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.138 DEBUG:boto:Method: GET 2024-03-04T16:17:17.138 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.138 DEBUG:boto:Data: 2024-03-04T16:17:17.139 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.139 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.139 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.139 DEBUG:boto:Params: {} 2024-03-04T16:17:17.139 DEBUG:boto:Token: None 2024-03-04T16:17:17.139 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/?versions 2024-03-04T16:17:17.139 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc= 2024-03-04T16:17:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc=', 'Content-Length': '0'} 2024-03-04T16:17:17.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c607e780c9c30842-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.184 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.184 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.184 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.185 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.185 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.185 DEBUG:boto:Method: GET 2024-03-04T16:17:17.185 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:17.185 DEBUG:boto:Data: 2024-03-04T16:17:17.185 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.185 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.185 DEBUG:boto:Params: {} 2024-03-04T16:17:17.185 DEBUG:boto:Token: None 2024-03-04T16:17:17.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/?versions 2024-03-04T16:17:17.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc= 2024-03-04T16:17:17.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OJDouY+K2+zyt18JBHLJ92FJ6Fc=', 'Content-Length': '0'} 2024-03-04T16:17:17.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198f1cea1f2ab727-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.188 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:17.189 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.189 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.189 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.189 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.189 DEBUG:boto:Method: GET 2024-03-04T16:17:17.189 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.189 DEBUG:boto:Data: 2024-03-04T16:17:17.189 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.189 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.190 DEBUG:boto:Params: {} 2024-03-04T16:17:17.190 DEBUG:boto:Token: None 2024-03-04T16:17:17.190 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw= 2024-03-04T16:17:17.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw=', 'Content-Length': '0'} 2024-03-04T16:17:17.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092ced0d944aa0d80-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.192 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.193 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.193 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.193 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.193 DEBUG:boto:Method: GET 2024-03-04T16:17:17.193 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.193 DEBUG:boto:Data: 2024-03-04T16:17:17.193 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.193 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.193 DEBUG:boto:Params: {} 2024-03-04T16:17:17.193 DEBUG:boto:Token: None 2024-03-04T16:17:17.194 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw= 2024-03-04T16:17:17.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw=', 'Content-Length': '0'} 2024-03-04T16:17:17.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a3ed7c41e93f6d0-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.196 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.196 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.196 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.196 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.196 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.196 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.196 DEBUG:boto:Data: 2024-03-04T16:17:17.196 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.196 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.196 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.197 DEBUG:boto:Params: {} 2024-03-04T16:17:17.197 DEBUG:boto:Token: None 2024-03-04T16:17:17.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.197 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FNy3LVuSi6aGoAE7YYdm3QZNLkQ= 2024-03-04T16:17:17.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FNy3LVuSi6aGoAE7YYdm3QZNLkQ=', 'Content-Length': '0'} 2024-03-04T16:17:17.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b50558b78ff45269-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.199 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.199 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.199 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.199 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.199 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.199 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.200 DEBUG:boto:Data: 2024-03-04T16:17:17.200 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.200 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.200 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.200 DEBUG:boto:Params: {} 2024-03-04T16:17:17.200 DEBUG:boto:Token: None 2024-03-04T16:17:17.200 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.200 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FNy3LVuSi6aGoAE7YYdm3QZNLkQ= 2024-03-04T16:17:17.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FNy3LVuSi6aGoAE7YYdm3QZNLkQ=', 'Content-Length': '0'} 2024-03-04T16:17:17.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000958d54ae0d31e93d-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:17.202 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.203 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.203 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.203 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.203 DEBUG:boto:Method: GET 2024-03-04T16:17:17.203 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.203 DEBUG:boto:Data: 2024-03-04T16:17:17.203 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.203 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.203 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.203 DEBUG:boto:Params: {} 2024-03-04T16:17:17.203 DEBUG:boto:Token: None 2024-03-04T16:17:17.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.204 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw= 2024-03-04T16:17:17.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw=', 'Content-Length': '0'} 2024-03-04T16:17:17.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fe167ebef582d58-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.206 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.206 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.206 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.206 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.206 DEBUG:boto:Method: GET 2024-03-04T16:17:17.206 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.206 DEBUG:boto:Data: 2024-03-04T16:17:17.206 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.206 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.206 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.206 DEBUG:boto:Params: {} 2024-03-04T16:17:17.207 DEBUG:boto:Token: None 2024-03-04T16:17:17.207 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:17.207 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw= 2024-03-04T16:17:17.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c9tFNT49D5+LNaOUkWeuwtgyKhw=', 'Content-Length': '0'} 2024-03-04T16:17:17.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d322d64e5bbf8740-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.209 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.209 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.209 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.209 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.209 DEBUG:boto:Method: GET 2024-03-04T16:17:17.209 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.209 DEBUG:boto:Data: 2024-03-04T16:17:17.209 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.209 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.209 DEBUG:boto:Params: {} 2024-03-04T16:17:17.210 DEBUG:boto:Token: None 2024-03-04T16:17:17.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o= 2024-03-04T16:17:17.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o=', 'Content-Length': '0'} 2024-03-04T16:17:17.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007829611468a550ec-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.212 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.212 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.213 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.213 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.213 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.213 DEBUG:boto:Method: GET 2024-03-04T16:17:17.213 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.213 DEBUG:boto:Data: 2024-03-04T16:17:17.213 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.213 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.213 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.214 DEBUG:boto:Params: {} 2024-03-04T16:17:17.214 DEBUG:boto:Token: None 2024-03-04T16:17:17.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.214 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o= 2024-03-04T16:17:17.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o=', 'Content-Length': '0'} 2024-03-04T16:17:17.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4ae432346f5303-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.260 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.260 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.261 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.261 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.261 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:17.261 DEBUG:boto:Data: 2024-03-04T16:17:17.261 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.261 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.261 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.261 DEBUG:boto:Params: {} 2024-03-04T16:17:17.261 DEBUG:boto:Token: None 2024-03-04T16:17:17.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj 2024-03-04T16:17:17.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q5EckAirdeyaStNZB9HCvcML7WI= 2024-03-04T16:17:17.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q5EckAirdeyaStNZB9HCvcML7WI=', 'Content-Length': '0'} 2024-03-04T16:17:17.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099df8c945f9109cd-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.264 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.264 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.264 DEBUG:boto:Method: HEAD 2024-03-04T16:17:17.264 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:17.264 DEBUG:boto:Data: 2024-03-04T16:17:17.264 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.264 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.264 DEBUG:boto:Params: {} 2024-03-04T16:17:17.265 DEBUG:boto:Token: None 2024-03-04T16:17:17.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj 2024-03-04T16:17:17.265 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q5EckAirdeyaStNZB9HCvcML7WI= 2024-03-04T16:17:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q5EckAirdeyaStNZB9HCvcML7WI=', 'Content-Length': '0'} 2024-03-04T16:17:17.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d35b3ca2c875a1e-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:17.267 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.267 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.267 DEBUG:boto:Method: GET 2024-03-04T16:17:17.267 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:17.268 DEBUG:boto:Data: 2024-03-04T16:17:17.268 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.268 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.268 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.268 DEBUG:boto:Params: {} 2024-03-04T16:17:17.268 DEBUG:boto:Token: None 2024-03-04T16:17:17.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj 2024-03-04T16:17:17.269 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HvoI6VS8tl4/LCa0l5fQQqHRePM= 2024-03-04T16:17:17.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HvoI6VS8tl4/LCa0l5fQQqHRePM=', 'Content-Length': '0'} 2024-03-04T16:17:17.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d003b44606684e1-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.271 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:17.271 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:17.271 DEBUG:boto:Method: GET 2024-03-04T16:17:17.272 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:17.272 DEBUG:boto:Data: 2024-03-04T16:17:17.272 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.272 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.272 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.272 DEBUG:boto:Params: {} 2024-03-04T16:17:17.272 DEBUG:boto:Token: None 2024-03-04T16:17:17.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/myobj 2024-03-04T16:17:17.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HvoI6VS8tl4/LCa0l5fQQqHRePM= 2024-03-04T16:17:17.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HvoI6VS8tl4/LCa0l5fQQqHRePM=', 'Content-Length': '0'} 2024-03-04T16:17:17.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e540225f96248b64-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.275 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.275 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.275 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.275 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.276 DEBUG:boto:Method: GET 2024-03-04T16:17:17.276 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.276 DEBUG:boto:Data: 2024-03-04T16:17:17.276 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:17:17.276 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.276 DEBUG:boto:Params: {} 2024-03-04T16:17:17.276 DEBUG:boto:Token: None 2024-03-04T16:17:17.276 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.276 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o= 2024-03-04T16:17:17.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o=', 'Content-Length': '0'} 2024-03-04T16:17:17.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4d5ab3e7c68386d-0065e5f40d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.279 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.279 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:17.279 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:17.280 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.280 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.280 DEBUG:boto:Method: GET 2024-03-04T16:17:17.280 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:17.280 DEBUG:boto:Data: 2024-03-04T16:17:17.280 DEBUG:boto:Headers: {} 2024-03-04T16:17:17.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:17.280 DEBUG:boto:Port: 8000 2024-03-04T16:17:17.280 DEBUG:boto:Params: {} 2024-03-04T16:17:17.280 DEBUG:boto:Token: None 2024-03-04T16:17:17.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:17 GMT /sbnvju-13/ 2024-03-04T16:17:17.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o= 2024-03-04T16:17:17.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eynLCLn7hyDl9xYcgh3n1OPMD5o=', 'Content-Length': '0'} 2024-03-04T16:17:17.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000484729e14f018ec5-0065e5f40d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:17.283 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:17.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-13 zones={test-zone2, test-zone1} 2024-03-04T16:17:17.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:17.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:17.284 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:17:17.344 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:17.344 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:17.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:17.397+0000 7f5cca739800 1 waiting to reach incremental sync.. 2024-03-04T16:17:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:22.399+0000 7f5cca739800 1 bucket sync caught up with source: 2024-03-04T16:17:22.400 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:17:22.400 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:17:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:22.399+0000 7f5cca739800 0 bucket checkpoint complete 2024-03-04T16:17:22.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-13 zones={test-zone3, test-zone1} 2024-03-04T16:17:22.423 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.423 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.423 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.423 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:22.423 DEBUG:boto:Data: 2024-03-04T16:17:22.423 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.423 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.424 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.424 DEBUG:boto:Params: {} 2024-03-04T16:17:22.424 DEBUG:boto:Token: None 2024-03-04T16:17:22.424 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.424 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kswrzaGnVcCQnFe+kT2wJdOhHhM= 2024-03-04T16:17:22.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kswrzaGnVcCQnFe+kT2wJdOhHhM=', 'Content-Length': '0'} 2024-03-04T16:17:22.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025682ce11f14c5b4-0065e5f412-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.427 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.427 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.427 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.428 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:17:22.428 DEBUG:boto:Data: 2024-03-04T16:17:22.428 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.428 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.428 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.428 DEBUG:boto:Params: {} 2024-03-04T16:17:22.428 DEBUG:boto:Token: None 2024-03-04T16:17:22.428 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.428 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kswrzaGnVcCQnFe+kT2wJdOhHhM= 2024-03-04T16:17:22.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kswrzaGnVcCQnFe+kT2wJdOhHhM=', 'Content-Length': '0'} 2024-03-04T16:17:22.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce7a0f7ca53893c4-0065e5f412-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.431 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:17:22.431 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.431 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.431 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.431 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.431 DEBUG:boto:Method: GET 2024-03-04T16:17:22.431 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.431 DEBUG:boto:Data: 2024-03-04T16:17:22.431 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.432 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.432 DEBUG:boto:Params: {} 2024-03-04T16:17:22.432 DEBUG:boto:Token: None 2024-03-04T16:17:22.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/?versions 2024-03-04T16:17:22.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo= 2024-03-04T16:17:22.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo=', 'Content-Length': '0'} 2024-03-04T16:17:22.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df35ab96fc555417-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.435 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.435 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:22.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:17:22.435 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.435 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.435 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.436 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.436 DEBUG:boto:Method: GET 2024-03-04T16:17:22.436 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.436 DEBUG:boto:Data: 2024-03-04T16:17:22.436 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.436 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.436 DEBUG:boto:Params: {} 2024-03-04T16:17:22.436 DEBUG:boto:Token: None 2024-03-04T16:17:22.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/?versions 2024-03-04T16:17:22.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo= 2024-03-04T16:17:22.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo=', 'Content-Length': '0'} 2024-03-04T16:17:22.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8c14a8e513416a-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.439 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.439 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:17:22.439 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.440 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.440 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.440 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.440 DEBUG:boto:Method: GET 2024-03-04T16:17:22.440 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.440 DEBUG:boto:Data: 2024-03-04T16:17:22.440 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.440 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.440 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.440 DEBUG:boto:Params: {} 2024-03-04T16:17:22.440 DEBUG:boto:Token: None 2024-03-04T16:17:22.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/?versions 2024-03-04T16:17:22.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo= 2024-03-04T16:17:22.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo=', 'Content-Length': '0'} 2024-03-04T16:17:22.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce83ef6f35e6df3-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.443 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.443 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.444 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.444 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.444 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.444 DEBUG:boto:Method: GET 2024-03-04T16:17:22.444 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:17:22.444 DEBUG:boto:Data: 2024-03-04T16:17:22.444 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.444 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.444 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.444 DEBUG:boto:Params: {} 2024-03-04T16:17:22.445 DEBUG:boto:Token: None 2024-03-04T16:17:22.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/?versions 2024-03-04T16:17:22.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo= 2024-03-04T16:17:22.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ye32/LGm1VVma4J329x30ZdASDo=', 'Content-Length': '0'} 2024-03-04T16:17:22.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b00b67b3b95f9626-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.447 DEBUG:boto:b'sbnvju-13999falsemyobjnulltrue2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:22.448 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.448 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.448 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.448 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.448 DEBUG:boto:Method: GET 2024-03-04T16:17:22.448 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.448 DEBUG:boto:Data: 2024-03-04T16:17:22.448 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.449 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.449 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.449 DEBUG:boto:Params: {} 2024-03-04T16:17:22.449 DEBUG:boto:Token: None 2024-03-04T16:17:22.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.449 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s= 2024-03-04T16:17:22.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s=', 'Content-Length': '0'} 2024-03-04T16:17:22.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaed4d53d72e1964-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.451 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.452 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.452 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.452 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.452 DEBUG:boto:Method: GET 2024-03-04T16:17:22.452 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.452 DEBUG:boto:Data: 2024-03-04T16:17:22.452 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.452 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.452 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.452 DEBUG:boto:Params: {} 2024-03-04T16:17:22.452 DEBUG:boto:Token: None 2024-03-04T16:17:22.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s= 2024-03-04T16:17:22.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s=', 'Content-Length': '0'} 2024-03-04T16:17:22.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001de264f1fd196441-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.455 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.455 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.455 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.455 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.455 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.455 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.455 DEBUG:boto:Data: 2024-03-04T16:17:22.456 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.456 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.456 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.456 DEBUG:boto:Params: {} 2024-03-04T16:17:22.456 DEBUG:boto:Token: None 2024-03-04T16:17:22.456 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.456 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xYF4pAcfq7EWyOskc67RwMLZ3ZY= 2024-03-04T16:17:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xYF4pAcfq7EWyOskc67RwMLZ3ZY=', 'Content-Length': '0'} 2024-03-04T16:17:22.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000606a72dad78cadcc-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.458 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.458 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.458 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.458 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.459 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.459 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.459 DEBUG:boto:Data: 2024-03-04T16:17:22.459 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.459 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.459 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.459 DEBUG:boto:Params: {} 2024-03-04T16:17:22.459 DEBUG:boto:Token: None 2024-03-04T16:17:22.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xYF4pAcfq7EWyOskc67RwMLZ3ZY= 2024-03-04T16:17:22.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xYF4pAcfq7EWyOskc67RwMLZ3ZY=', 'Content-Length': '0'} 2024-03-04T16:17:22.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047bba62942465306-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:22.461 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.461 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.462 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.462 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.462 DEBUG:boto:Method: GET 2024-03-04T16:17:22.462 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.462 DEBUG:boto:Data: 2024-03-04T16:17:22.462 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.462 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.462 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.462 DEBUG:boto:Params: {} 2024-03-04T16:17:22.462 DEBUG:boto:Token: None 2024-03-04T16:17:22.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.463 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s= 2024-03-04T16:17:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s=', 'Content-Length': '0'} 2024-03-04T16:17:22.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c52e29359e71a4b5-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.465 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.465 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.465 DEBUG:boto:path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.465 DEBUG:boto:auth_path=/sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.465 DEBUG:boto:Method: GET 2024-03-04T16:17:22.465 DEBUG:boto:Path: /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.465 DEBUG:boto:Data: 2024-03-04T16:17:22.465 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.465 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.465 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.465 DEBUG:boto:Params: {} 2024-03-04T16:17:22.466 DEBUG:boto:Token: None 2024-03-04T16:17:22.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj?versionId=null 2024-03-04T16:17:22.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s= 2024-03-04T16:17:22.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bC1xyOq6wkSTSr3zcpjEVfSnH2s=', 'Content-Length': '0'} 2024-03-04T16:17:22.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007028d8b4d2b4fe9c-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.468 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.468 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.468 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.468 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.468 DEBUG:boto:Method: GET 2024-03-04T16:17:22.468 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.468 DEBUG:boto:Data: 2024-03-04T16:17:22.468 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.468 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.468 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.468 DEBUG:boto:Params: {} 2024-03-04T16:17:22.469 DEBUG:boto:Token: None 2024-03-04T16:17:22.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.469 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE= 2024-03-04T16:17:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE=', 'Content-Length': '0'} 2024-03-04T16:17:22.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c111b141ddaa31-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.471 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.472 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.472 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.472 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.472 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.472 DEBUG:boto:Method: GET 2024-03-04T16:17:22.472 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.472 DEBUG:boto:Data: 2024-03-04T16:17:22.472 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.472 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.472 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.472 DEBUG:boto:Params: {} 2024-03-04T16:17:22.473 DEBUG:boto:Token: None 2024-03-04T16:17:22.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE= 2024-03-04T16:17:22.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE=', 'Content-Length': '0'} 2024-03-04T16:17:22.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f1073d166213f4-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.475 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.476 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.476 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.476 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.476 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:22.476 DEBUG:boto:Data: 2024-03-04T16:17:22.476 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.476 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.476 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.476 DEBUG:boto:Params: {} 2024-03-04T16:17:22.476 DEBUG:boto:Token: None 2024-03-04T16:17:22.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj 2024-03-04T16:17:22.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o9B+rmJZDJI2eRg+Zc/rxvAhp8s= 2024-03-04T16:17:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o9B+rmJZDJI2eRg+Zc/rxvAhp8s=', 'Content-Length': '0'} 2024-03-04T16:17:22.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007423af855523cc76-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.479 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.479 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.479 DEBUG:boto:Method: HEAD 2024-03-04T16:17:22.479 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:22.479 DEBUG:boto:Data: 2024-03-04T16:17:22.479 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.479 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.479 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.479 DEBUG:boto:Params: {} 2024-03-04T16:17:22.479 DEBUG:boto:Token: None 2024-03-04T16:17:22.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj 2024-03-04T16:17:22.480 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o9B+rmJZDJI2eRg+Zc/rxvAhp8s= 2024-03-04T16:17:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o9B+rmJZDJI2eRg+Zc/rxvAhp8s=', 'Content-Length': '0'} 2024-03-04T16:17:22.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4bee4f5ce5b8c3b-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:17:22.482 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.482 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.482 DEBUG:boto:Method: GET 2024-03-04T16:17:22.482 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:22.482 DEBUG:boto:Data: 2024-03-04T16:17:22.482 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.482 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.482 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.482 DEBUG:boto:Params: {} 2024-03-04T16:17:22.482 DEBUG:boto:Token: None 2024-03-04T16:17:22.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj 2024-03-04T16:17:22.483 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YBQHCz93igCW3O9bYns28RegVKg= 2024-03-04T16:17:22.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YBQHCz93igCW3O9bYns28RegVKg=', 'Content-Length': '0'} 2024-03-04T16:17:22.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-13:fc294220-5038-4f25-aeee-cac2c5838729.4341.22'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006903aa2dbb262d9c-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.528 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:17:22.528 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:17:22.528 DEBUG:boto:Method: GET 2024-03-04T16:17:22.528 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:17:22.528 DEBUG:boto:Data: 2024-03-04T16:17:22.528 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.529 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.529 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.529 DEBUG:boto:Params: {} 2024-03-04T16:17:22.529 DEBUG:boto:Token: None 2024-03-04T16:17:22.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/myobj 2024-03-04T16:17:22.529 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YBQHCz93igCW3O9bYns28RegVKg= 2024-03-04T16:17:22.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YBQHCz93igCW3O9bYns28RegVKg=', 'Content-Length': '0'} 2024-03-04T16:17:22.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065ac49158d34d13f-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.531 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.531 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.531 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.532 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.532 DEBUG:boto:Method: GET 2024-03-04T16:17:22.532 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.532 DEBUG:boto:Data: 2024-03-04T16:17:22.532 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.532 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:17:22.532 DEBUG:boto:Port: 8001 2024-03-04T16:17:22.532 DEBUG:boto:Params: {} 2024-03-04T16:17:22.532 DEBUG:boto:Token: None 2024-03-04T16:17:22.532 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.532 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE= 2024-03-04T16:17:22.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE=', 'Content-Length': '0'} 2024-03-04T16:17:22.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a93bd068870bd5a-0065e5f412-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.576 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.576 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:17:22.576 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:17:22.577 DEBUG:boto:path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.577 DEBUG:boto:auth_path=/sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.577 DEBUG:boto:Method: GET 2024-03-04T16:17:22.577 DEBUG:boto:Path: /sbnvju-13/?prefix=myobj 2024-03-04T16:17:22.577 DEBUG:boto:Data: 2024-03-04T16:17:22.577 DEBUG:boto:Headers: {} 2024-03-04T16:17:22.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:17:22.577 DEBUG:boto:Port: 8000 2024-03-04T16:17:22.577 DEBUG:boto:Params: {} 2024-03-04T16:17:22.577 DEBUG:boto:Token: None 2024-03-04T16:17:22.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:17:22 GMT /sbnvju-13/ 2024-03-04T16:17:22.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE= 2024-03-04T16:17:22.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:17:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:STXEF5oDdArEoj3X7DbRGkIOECE=', 'Content-Length': '0'} 2024-03-04T16:17:22.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c87f35a30d3b4843-0065e5f412-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:17:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:17:22.580 DEBUG:boto:b'sbnvju-13myobj1000falsemyobj2024-03-04T16:17:01.584Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:17:22.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-13 zones={test-zone3, test-zone1} 2024-03-04T16:17:22.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:22.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:17:22.581 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:17:22.638 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:17:22.638 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:17:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:22.692+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:27.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:27.692+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:32.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:32.692+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:37.693+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:42.694+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:47.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:47.694+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:52.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:52.695+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:17:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:17:57.696+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:18:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:02.696+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:18:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:07.697+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:18:12.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:12.698+0000 7f99e1930800 1 waiting to reach incremental sync.. 2024-03-04T16:18:17.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:17.700+0000 7f99e1930800 1 bucket sync caught up with source: 2024-03-04T16:18:17.701 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:17.701 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:17.700+0000 7f99e1930800 0 bucket checkpoint complete 2024-03-04T16:18:17.725 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-14 zones={test-zone1, test-zone2} 2024-03-04T16:18:17.726 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.726 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.726 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.726 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:17.726 DEBUG:boto:Data: 2024-03-04T16:18:17.726 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.726 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.726 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.726 DEBUG:boto:Params: {} 2024-03-04T16:18:17.727 DEBUG:boto:Token: None 2024-03-04T16:18:17.727 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hMNoBiD+07CIAw3cD7vn33wq+TA= 2024-03-04T16:18:17.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hMNoBiD+07CIAw3cD7vn33wq+TA=', 'Content-Length': '0'} 2024-03-04T16:18:17.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095c47e925793f303-0065e5f449-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.730 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.731 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.731 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.731 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:17.731 DEBUG:boto:Data: 2024-03-04T16:18:17.731 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.731 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.731 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.731 DEBUG:boto:Params: {} 2024-03-04T16:18:17.731 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:18:17.731 DEBUG:boto:Token: None 2024-03-04T16:18:17.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.732 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hMNoBiD+07CIAw3cD7vn33wq+TA= 2024-03-04T16:18:17.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hMNoBiD+07CIAw3cD7vn33wq+TA=', 'Content-Length': '0'} 2024-03-04T16:18:17.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003699c785f98dfa09-0065e5f449-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.737 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:17.737 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.737 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.738 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.738 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.738 DEBUG:boto:Method: GET 2024-03-04T16:18:17.738 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.738 DEBUG:boto:Data: 2024-03-04T16:18:17.738 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.738 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.738 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.738 DEBUG:boto:Params: {} 2024-03-04T16:18:17.738 DEBUG:boto:Token: None 2024-03-04T16:18:17.738 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/?versions 2024-03-04T16:18:17.739 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I= 2024-03-04T16:18:17.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I=', 'Content-Length': '0'} 2024-03-04T16:18:17.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab7649af17fb90a-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.741 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.742 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:17.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:17.743 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.743 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.743 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.743 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.743 DEBUG:boto:Method: GET 2024-03-04T16:18:17.743 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.743 DEBUG:boto:Data: 2024-03-04T16:18:17.743 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.743 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.743 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.743 DEBUG:boto:Params: {} 2024-03-04T16:18:17.743 DEBUG:boto:Token: None 2024-03-04T16:18:17.744 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/?versions 2024-03-04T16:18:17.744 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I= 2024-03-04T16:18:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I=', 'Content-Length': '0'} 2024-03-04T16:18:17.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008873d82bb91d1694-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.746 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.747 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:17.747 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.747 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.747 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.747 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.747 DEBUG:boto:Method: GET 2024-03-04T16:18:17.747 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.747 DEBUG:boto:Data: 2024-03-04T16:18:17.747 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.748 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.748 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.748 DEBUG:boto:Params: {} 2024-03-04T16:18:17.748 DEBUG:boto:Token: None 2024-03-04T16:18:17.748 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/?versions 2024-03-04T16:18:17.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I= 2024-03-04T16:18:17.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I=', 'Content-Length': '0'} 2024-03-04T16:18:17.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f68b590a12dba9ea-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.750 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.751 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.751 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.751 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.751 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.751 DEBUG:boto:Method: GET 2024-03-04T16:18:17.751 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:17.751 DEBUG:boto:Data: 2024-03-04T16:18:17.751 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.751 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.752 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.752 DEBUG:boto:Params: {} 2024-03-04T16:18:17.752 DEBUG:boto:Token: None 2024-03-04T16:18:17.752 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/?versions 2024-03-04T16:18:17.752 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I= 2024-03-04T16:18:17.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2ofWIhXgbL6p70ahRoWAmtztC8I=', 'Content-Length': '0'} 2024-03-04T16:18:17.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2baaed0716ea269-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.754 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:17.755 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.755 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.755 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.755 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.755 DEBUG:boto:Method: GET 2024-03-04T16:18:17.755 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.755 DEBUG:boto:Data: 2024-03-04T16:18:17.756 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.756 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.756 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.756 DEBUG:boto:Params: {} 2024-03-04T16:18:17.756 DEBUG:boto:Token: None 2024-03-04T16:18:17.756 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.756 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE= 2024-03-04T16:18:17.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE=', 'Content-Length': '0'} 2024-03-04T16:18:17.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e193a70243280618-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.759 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.759 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.759 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.759 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.759 DEBUG:boto:Method: GET 2024-03-04T16:18:17.759 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.759 DEBUG:boto:Data: 2024-03-04T16:18:17.759 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.759 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.759 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.759 DEBUG:boto:Params: {} 2024-03-04T16:18:17.759 DEBUG:boto:Token: None 2024-03-04T16:18:17.760 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.760 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE= 2024-03-04T16:18:17.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE=', 'Content-Length': '0'} 2024-03-04T16:18:17.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000726d05a9dd6b5ec1-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.762 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.762 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.762 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.762 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.763 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.763 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.763 DEBUG:boto:Data: 2024-03-04T16:18:17.763 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.763 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.763 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.763 DEBUG:boto:Params: {} 2024-03-04T16:18:17.763 DEBUG:boto:Token: None 2024-03-04T16:18:17.763 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.764 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VRQar3YYLAf4XZhpJS56faKrm0s= 2024-03-04T16:18:17.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VRQar3YYLAf4XZhpJS56faKrm0s=', 'Content-Length': '0'} 2024-03-04T16:18:17.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000608512fbff301c17-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.766 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.766 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.766 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.766 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.766 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.766 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.766 DEBUG:boto:Data: 2024-03-04T16:18:17.767 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.767 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.767 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.767 DEBUG:boto:Params: {} 2024-03-04T16:18:17.767 DEBUG:boto:Token: None 2024-03-04T16:18:17.767 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.767 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VRQar3YYLAf4XZhpJS56faKrm0s= 2024-03-04T16:18:17.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VRQar3YYLAf4XZhpJS56faKrm0s=', 'Content-Length': '0'} 2024-03-04T16:18:17.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000902aa995011ea37d-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:17.769 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.769 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.769 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.770 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.770 DEBUG:boto:Method: GET 2024-03-04T16:18:17.770 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.770 DEBUG:boto:Data: 2024-03-04T16:18:17.770 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.770 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.770 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.770 DEBUG:boto:Params: {} 2024-03-04T16:18:17.770 DEBUG:boto:Token: None 2024-03-04T16:18:17.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.771 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE= 2024-03-04T16:18:17.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE=', 'Content-Length': '0'} 2024-03-04T16:18:17.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f648ce1ef28940b8-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.773 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.773 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.773 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.773 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.773 DEBUG:boto:Method: GET 2024-03-04T16:18:17.773 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.773 DEBUG:boto:Data: 2024-03-04T16:18:17.773 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.773 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.773 DEBUG:boto:Params: {} 2024-03-04T16:18:17.774 DEBUG:boto:Token: None 2024-03-04T16:18:17.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:17.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE= 2024-03-04T16:18:17.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G8K7QcYPrjM5s3PJfzK8om304IE=', 'Content-Length': '0'} 2024-03-04T16:18:17.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feefb991dd4531d6-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.777 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.777 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.777 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.778 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.778 DEBUG:boto:Method: GET 2024-03-04T16:18:17.778 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.778 DEBUG:boto:Data: 2024-03-04T16:18:17.778 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.778 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.778 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.778 DEBUG:boto:Params: {} 2024-03-04T16:18:17.778 DEBUG:boto:Token: None 2024-03-04T16:18:17.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno= 2024-03-04T16:18:17.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno=', 'Content-Length': '0'} 2024-03-04T16:18:17.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059503b465f879816-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.824 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.825 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.825 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.825 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.825 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.825 DEBUG:boto:Method: GET 2024-03-04T16:18:17.825 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.825 DEBUG:boto:Data: 2024-03-04T16:18:17.825 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.825 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.825 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.825 DEBUG:boto:Params: {} 2024-03-04T16:18:17.826 DEBUG:boto:Token: None 2024-03-04T16:18:17.826 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.826 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno= 2024-03-04T16:18:17.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno=', 'Content-Length': '0'} 2024-03-04T16:18:17.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c64092db59d3e025-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.829 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.829 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.829 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.829 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.829 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:17.829 DEBUG:boto:Data: 2024-03-04T16:18:17.829 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.830 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.830 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.830 DEBUG:boto:Params: {} 2024-03-04T16:18:17.830 DEBUG:boto:Token: None 2024-03-04T16:18:17.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj 2024-03-04T16:18:17.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KQhrZ+C/FrhBImB/NHZomtKtmpM= 2024-03-04T16:18:17.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KQhrZ+C/FrhBImB/NHZomtKtmpM=', 'Content-Length': '0'} 2024-03-04T16:18:17.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b9492ef804894b-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.832 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.832 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.832 DEBUG:boto:Method: HEAD 2024-03-04T16:18:17.832 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:17.833 DEBUG:boto:Data: 2024-03-04T16:18:17.833 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.833 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.833 DEBUG:boto:Params: {} 2024-03-04T16:18:17.833 DEBUG:boto:Token: None 2024-03-04T16:18:17.833 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj 2024-03-04T16:18:17.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KQhrZ+C/FrhBImB/NHZomtKtmpM= 2024-03-04T16:18:17.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KQhrZ+C/FrhBImB/NHZomtKtmpM=', 'Content-Length': '0'} 2024-03-04T16:18:17.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025ce5c3f86dbd2ee-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:17.836 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.836 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.836 DEBUG:boto:Method: GET 2024-03-04T16:18:17.836 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:17.836 DEBUG:boto:Data: 2024-03-04T16:18:17.836 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.836 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.836 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.836 DEBUG:boto:Params: {} 2024-03-04T16:18:17.836 DEBUG:boto:Token: None 2024-03-04T16:18:17.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj 2024-03-04T16:18:17.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Zlp8GwZWrW8mMd2ceO6XPM9dEhc= 2024-03-04T16:18:17.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Zlp8GwZWrW8mMd2ceO6XPM9dEhc=', 'Content-Length': '0'} 2024-03-04T16:18:17.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fe3a01892ba291f-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.880 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:17.880 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:17.880 DEBUG:boto:Method: GET 2024-03-04T16:18:17.880 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:17.880 DEBUG:boto:Data: 2024-03-04T16:18:17.881 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.881 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.881 DEBUG:boto:Params: {} 2024-03-04T16:18:17.881 DEBUG:boto:Token: None 2024-03-04T16:18:17.881 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/myobj 2024-03-04T16:18:17.881 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Zlp8GwZWrW8mMd2ceO6XPM9dEhc= 2024-03-04T16:18:17.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Zlp8GwZWrW8mMd2ceO6XPM9dEhc=', 'Content-Length': '0'} 2024-03-04T16:18:17.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c711881d03bb8c10-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.883 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.883 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.884 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.884 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.884 DEBUG:boto:Method: GET 2024-03-04T16:18:17.884 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.884 DEBUG:boto:Data: 2024-03-04T16:18:17.884 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.884 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:17.884 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.884 DEBUG:boto:Params: {} 2024-03-04T16:18:17.884 DEBUG:boto:Token: None 2024-03-04T16:18:17.885 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.885 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno= 2024-03-04T16:18:17.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno=', 'Content-Length': '0'} 2024-03-04T16:18:17.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c21f75e23685254-0065e5f449-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.887 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.888 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:17.888 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:17.888 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.888 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.888 DEBUG:boto:Method: GET 2024-03-04T16:18:17.888 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:17.888 DEBUG:boto:Data: 2024-03-04T16:18:17.888 DEBUG:boto:Headers: {} 2024-03-04T16:18:17.888 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:17.888 DEBUG:boto:Port: 8000 2024-03-04T16:18:17.888 DEBUG:boto:Params: {} 2024-03-04T16:18:17.889 DEBUG:boto:Token: None 2024-03-04T16:18:17.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:17 GMT /sbnvju-14/ 2024-03-04T16:18:17.889 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno= 2024-03-04T16:18:17.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jupPA+sht+0hGR0tR8ENHgJSVno=', 'Content-Length': '0'} 2024-03-04T16:18:17.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc3a971104792a8-0065e5f449-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:17.892 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:17.892 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-14 zones={test-zone1, test-zone2} 2024-03-04T16:18:17.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:17.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:17.893 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:18.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:18.009+0000 7f2d4f42e800 1 bucket sync caught up with source: 2024-03-04T16:18:18.010 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:18.010 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:18.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:18.009+0000 7f2d4f42e800 0 bucket checkpoint complete 2024-03-04T16:18:18.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-14 zones={test-zone3, test-zone2} 2024-03-04T16:18:18.029 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.029 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.029 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.030 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:18.030 DEBUG:boto:Data: 2024-03-04T16:18:18.030 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.030 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.030 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.030 DEBUG:boto:Params: {} 2024-03-04T16:18:18.030 DEBUG:boto:Token: None 2024-03-04T16:18:18.030 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iXO+Nq2EMB8Iu6jmXNh8mVYNeY= 2024-03-04T16:18:18.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iXO+Nq2EMB8Iu6jmXNh8mVYNeY=', 'Content-Length': '0'} 2024-03-04T16:18:18.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7533b5ed57433d1-0065e5f44a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.033 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.033 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.033 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.034 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:18.034 DEBUG:boto:Data: 2024-03-04T16:18:18.034 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.034 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.034 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.034 DEBUG:boto:Params: {} 2024-03-04T16:18:18.034 DEBUG:boto:Token: None 2024-03-04T16:18:18.034 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iXO+Nq2EMB8Iu6jmXNh8mVYNeY= 2024-03-04T16:18:18.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iXO+Nq2EMB8Iu6jmXNh8mVYNeY=', 'Content-Length': '0'} 2024-03-04T16:18:18.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045507dac3994a569-0065e5f44a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:18.037 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.037 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.037 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.038 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.038 DEBUG:boto:Method: GET 2024-03-04T16:18:18.038 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.038 DEBUG:boto:Data: 2024-03-04T16:18:18.038 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.038 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.038 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.038 DEBUG:boto:Params: {} 2024-03-04T16:18:18.038 DEBUG:boto:Token: None 2024-03-04T16:18:18.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/?versions 2024-03-04T16:18:18.039 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI= 2024-03-04T16:18:18.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI=', 'Content-Length': '0'} 2024-03-04T16:18:18.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137453e1a68cf949-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.041 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.041 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:18.042 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.042 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.042 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.042 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.042 DEBUG:boto:Method: GET 2024-03-04T16:18:18.042 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.042 DEBUG:boto:Data: 2024-03-04T16:18:18.042 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.042 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.042 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.042 DEBUG:boto:Params: {} 2024-03-04T16:18:18.043 DEBUG:boto:Token: None 2024-03-04T16:18:18.043 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/?versions 2024-03-04T16:18:18.043 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI= 2024-03-04T16:18:18.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI=', 'Content-Length': '0'} 2024-03-04T16:18:18.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf14eb01907df84-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.046 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.046 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.046 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.046 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.046 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.047 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.047 DEBUG:boto:Method: GET 2024-03-04T16:18:18.047 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.047 DEBUG:boto:Data: 2024-03-04T16:18:18.047 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.047 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.047 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.047 DEBUG:boto:Params: {} 2024-03-04T16:18:18.047 DEBUG:boto:Token: None 2024-03-04T16:18:18.047 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/?versions 2024-03-04T16:18:18.047 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI= 2024-03-04T16:18:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI=', 'Content-Length': '0'} 2024-03-04T16:18:18.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f28179ab5ab3b8f5-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.050 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.050 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.050 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.050 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.050 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.050 DEBUG:boto:Method: GET 2024-03-04T16:18:18.051 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:18.051 DEBUG:boto:Data: 2024-03-04T16:18:18.051 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.051 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.051 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.051 DEBUG:boto:Params: {} 2024-03-04T16:18:18.051 DEBUG:boto:Token: None 2024-03-04T16:18:18.051 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/?versions 2024-03-04T16:18:18.051 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI= 2024-03-04T16:18:18.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rT1VUe3A2E5N3nymnKOYVK5kIeI=', 'Content-Length': '0'} 2024-03-04T16:18:18.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753c184e23d0b10a-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.054 DEBUG:boto:b'sbnvju-14999falsemyobjnulltrue2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.055 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.055 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.055 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.055 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.055 DEBUG:boto:Method: GET 2024-03-04T16:18:18.055 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.055 DEBUG:boto:Data: 2024-03-04T16:18:18.055 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.055 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.056 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.056 DEBUG:boto:Params: {} 2024-03-04T16:18:18.056 DEBUG:boto:Token: None 2024-03-04T16:18:18.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.056 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ= 2024-03-04T16:18:18.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ=', 'Content-Length': '0'} 2024-03-04T16:18:18.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f9c49629cbb41b1-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.058 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.058 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.058 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.058 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.058 DEBUG:boto:Method: GET 2024-03-04T16:18:18.058 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.058 DEBUG:boto:Data: 2024-03-04T16:18:18.058 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.058 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.059 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.059 DEBUG:boto:Params: {} 2024-03-04T16:18:18.059 DEBUG:boto:Token: None 2024-03-04T16:18:18.059 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.059 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ= 2024-03-04T16:18:18.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ=', 'Content-Length': '0'} 2024-03-04T16:18:18.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb35e54231e819f1-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.061 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.061 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.061 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.061 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.061 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.062 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.062 DEBUG:boto:Data: 2024-03-04T16:18:18.062 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.062 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.062 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.062 DEBUG:boto:Params: {} 2024-03-04T16:18:18.062 DEBUG:boto:Token: None 2024-03-04T16:18:18.062 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:b4Tl4eW7CDa83DGyyz8V0Zvjqcw= 2024-03-04T16:18:18.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b4Tl4eW7CDa83DGyyz8V0Zvjqcw=', 'Content-Length': '0'} 2024-03-04T16:18:18.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0c52f43e032ecb3-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.064 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.064 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.064 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.064 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.064 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.065 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.065 DEBUG:boto:Data: 2024-03-04T16:18:18.065 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.065 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.065 DEBUG:boto:Params: {} 2024-03-04T16:18:18.065 DEBUG:boto:Token: None 2024-03-04T16:18:18.065 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:b4Tl4eW7CDa83DGyyz8V0Zvjqcw= 2024-03-04T16:18:18.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b4Tl4eW7CDa83DGyyz8V0Zvjqcw=', 'Content-Length': '0'} 2024-03-04T16:18:18.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b3ed9f1a9b243f6-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.067 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.068 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.068 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.068 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.068 DEBUG:boto:Method: GET 2024-03-04T16:18:18.068 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.068 DEBUG:boto:Data: 2024-03-04T16:18:18.068 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.068 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.068 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.068 DEBUG:boto:Params: {} 2024-03-04T16:18:18.068 DEBUG:boto:Token: None 2024-03-04T16:18:18.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ= 2024-03-04T16:18:18.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ=', 'Content-Length': '0'} 2024-03-04T16:18:18.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000043d09dd75b17886-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.071 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.071 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.071 DEBUG:boto:path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.071 DEBUG:boto:auth_path=/sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.071 DEBUG:boto:Method: GET 2024-03-04T16:18:18.071 DEBUG:boto:Path: /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.071 DEBUG:boto:Data: 2024-03-04T16:18:18.071 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.071 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.071 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.071 DEBUG:boto:Params: {} 2024-03-04T16:18:18.071 DEBUG:boto:Token: None 2024-03-04T16:18:18.072 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj?versionId=null 2024-03-04T16:18:18.072 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ= 2024-03-04T16:18:18.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qRNAq82wOpD1JhWLb8ohttJ0rPQ=', 'Content-Length': '0'} 2024-03-04T16:18:18.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000353a7569ccfbe889-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.116 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.116 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.116 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.116 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.116 DEBUG:boto:Method: GET 2024-03-04T16:18:18.117 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.117 DEBUG:boto:Data: 2024-03-04T16:18:18.117 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.117 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.117 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.117 DEBUG:boto:Params: {} 2024-03-04T16:18:18.117 DEBUG:boto:Token: None 2024-03-04T16:18:18.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0= 2024-03-04T16:18:18.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0=', 'Content-Length': '0'} 2024-03-04T16:18:18.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bd270113a52e6c-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.120 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.120 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.120 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.120 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.120 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.120 DEBUG:boto:Method: GET 2024-03-04T16:18:18.120 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.121 DEBUG:boto:Data: 2024-03-04T16:18:18.121 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.121 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.121 DEBUG:boto:Params: {} 2024-03-04T16:18:18.121 DEBUG:boto:Token: None 2024-03-04T16:18:18.121 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.121 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0= 2024-03-04T16:18:18.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0=', 'Content-Length': '0'} 2024-03-04T16:18:18.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000012b7cd99778ba0-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.124 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.124 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.124 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.125 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.125 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:18.125 DEBUG:boto:Data: 2024-03-04T16:18:18.125 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.125 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.125 DEBUG:boto:Params: {} 2024-03-04T16:18:18.125 DEBUG:boto:Token: None 2024-03-04T16:18:18.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj 2024-03-04T16:18:18.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VgSEKOjY/J4tEyGRvgy8HJdT4ao= 2024-03-04T16:18:18.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VgSEKOjY/J4tEyGRvgy8HJdT4ao=', 'Content-Length': '0'} 2024-03-04T16:18:18.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f5a5c9e0dc1ade-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.128 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.128 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.128 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.128 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:18.128 DEBUG:boto:Data: 2024-03-04T16:18:18.128 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.128 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.128 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.128 DEBUG:boto:Params: {} 2024-03-04T16:18:18.128 DEBUG:boto:Token: None 2024-03-04T16:18:18.129 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj 2024-03-04T16:18:18.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VgSEKOjY/J4tEyGRvgy8HJdT4ao= 2024-03-04T16:18:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VgSEKOjY/J4tEyGRvgy8HJdT4ao=', 'Content-Length': '0'} 2024-03-04T16:18:18.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d2e16bae4f69f69-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.131 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.131 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.131 DEBUG:boto:Method: GET 2024-03-04T16:18:18.131 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:18.131 DEBUG:boto:Data: 2024-03-04T16:18:18.131 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.131 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.131 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.132 DEBUG:boto:Params: {} 2024-03-04T16:18:18.132 DEBUG:boto:Token: None 2024-03-04T16:18:18.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj 2024-03-04T16:18:18.132 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AzD0lNWIjwVi1KC67R8i4uAO/a8= 2024-03-04T16:18:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AzD0lNWIjwVi1KC67R8i4uAO/a8=', 'Content-Length': '0'} 2024-03-04T16:18:18.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-14:fc294220-5038-4f25-aeee-cac2c5838729.4341.23'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000739a9dd00974ccf2-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.176 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:18.176 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:18.176 DEBUG:boto:Method: GET 2024-03-04T16:18:18.176 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:18.176 DEBUG:boto:Data: 2024-03-04T16:18:18.176 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.177 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.177 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.177 DEBUG:boto:Params: {} 2024-03-04T16:18:18.177 DEBUG:boto:Token: None 2024-03-04T16:18:18.177 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/myobj 2024-03-04T16:18:18.177 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AzD0lNWIjwVi1KC67R8i4uAO/a8= 2024-03-04T16:18:18.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AzD0lNWIjwVi1KC67R8i4uAO/a8=', 'Content-Length': '0'} 2024-03-04T16:18:18.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a01b86e5ddff2041-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.179 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.180 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.180 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.180 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.180 DEBUG:boto:Method: GET 2024-03-04T16:18:18.180 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.180 DEBUG:boto:Data: 2024-03-04T16:18:18.180 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.180 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.180 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.180 DEBUG:boto:Params: {} 2024-03-04T16:18:18.180 DEBUG:boto:Token: None 2024-03-04T16:18:18.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.181 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0= 2024-03-04T16:18:18.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0=', 'Content-Length': '0'} 2024-03-04T16:18:18.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a5b3dd18f3d086-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.228 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.229 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:18.229 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:18.229 DEBUG:boto:path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.229 DEBUG:boto:auth_path=/sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.229 DEBUG:boto:Method: GET 2024-03-04T16:18:18.229 DEBUG:boto:Path: /sbnvju-14/?prefix=myobj 2024-03-04T16:18:18.229 DEBUG:boto:Data: 2024-03-04T16:18:18.229 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.229 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.229 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.229 DEBUG:boto:Params: {} 2024-03-04T16:18:18.230 DEBUG:boto:Token: None 2024-03-04T16:18:18.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-14/ 2024-03-04T16:18:18.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0= 2024-03-04T16:18:18.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cu8KVnuYal6SG437Y28W63RQOd0=', 'Content-Length': '0'} 2024-03-04T16:18:18.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340a7e87e9c8e3fd-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.233 DEBUG:boto:b'sbnvju-14myobj1000falsemyobj2024-03-04T16:17:01.636Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.233 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-14 zones={test-zone3, test-zone2} 2024-03-04T16:18:18.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.234 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:18.341+0000 7fd9ca12f800 1 bucket sync caught up with source: 2024-03-04T16:18:18.341 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:18.341 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:18.341+0000 7fd9ca12f800 0 bucket checkpoint complete 2024-03-04T16:18:18.360 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-15 zones={test-zone1, test-zone3} 2024-03-04T16:18:18.360 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.360 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.361 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.361 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:18:18.361 DEBUG:boto:Data: 2024-03-04T16:18:18.361 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.361 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.361 DEBUG:boto:Params: {} 2024-03-04T16:18:18.361 DEBUG:boto:Token: None 2024-03-04T16:18:18.361 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw= 2024-03-04T16:18:18.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw=', 'Content-Length': '0'} 2024-03-04T16:18:18.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006abdd23bfd752f13-0065e5f44a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.364 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.364 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.364 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.364 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:18:18.364 DEBUG:boto:Data: 2024-03-04T16:18:18.365 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.365 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.365 DEBUG:boto:Params: {} 2024-03-04T16:18:18.365 DEBUG:boto:Token: None 2024-03-04T16:18:18.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw= 2024-03-04T16:18:18.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw=', 'Content-Length': '0'} 2024-03-04T16:18:18.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e69e32a19dadb0b-0065e5f44a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:18.368 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.368 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.368 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.368 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.368 DEBUG:boto:Method: GET 2024-03-04T16:18:18.368 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.368 DEBUG:boto:Data: 2024-03-04T16:18:18.368 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.368 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.368 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.368 DEBUG:boto:Params: {} 2024-03-04T16:18:18.369 DEBUG:boto:Token: None 2024-03-04T16:18:18.369 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.369 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d031523ceb5a62ac-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.371 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.371 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:18.372 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.372 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.372 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.372 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.372 DEBUG:boto:Method: GET 2024-03-04T16:18:18.372 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.372 DEBUG:boto:Data: 2024-03-04T16:18:18.372 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.372 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.372 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.372 DEBUG:boto:Params: {} 2024-03-04T16:18:18.373 DEBUG:boto:Token: None 2024-03-04T16:18:18.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcc06f0a08760c48-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.375 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.375 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.376 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.376 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.376 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.376 DEBUG:boto:Method: GET 2024-03-04T16:18:18.376 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.376 DEBUG:boto:Data: 2024-03-04T16:18:18.376 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.376 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.376 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.376 DEBUG:boto:Params: {} 2024-03-04T16:18:18.376 DEBUG:boto:Token: None 2024-03-04T16:18:18.377 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.377 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c070b71b4e5035a-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.379 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.379 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.380 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.380 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.380 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.380 DEBUG:boto:Method: GET 2024-03-04T16:18:18.380 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.380 DEBUG:boto:Data: 2024-03-04T16:18:18.380 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.380 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.380 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.380 DEBUG:boto:Params: {} 2024-03-04T16:18:18.381 DEBUG:boto:Token: None 2024-03-04T16:18:18.381 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f17bf695bd07d332-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.383 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.384 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.384 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.384 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.384 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.384 DEBUG:boto:Method: GET 2024-03-04T16:18:18.384 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.384 DEBUG:boto:Data: 2024-03-04T16:18:18.384 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.385 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.385 DEBUG:boto:Params: {} 2024-03-04T16:18:18.385 DEBUG:boto:Token: None 2024-03-04T16:18:18.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006587a066404a3fde-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.387 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.387 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.387 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.387 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.388 DEBUG:boto:Method: GET 2024-03-04T16:18:18.388 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.388 DEBUG:boto:Data: 2024-03-04T16:18:18.388 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.388 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.388 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.388 DEBUG:boto:Params: {} 2024-03-04T16:18:18.388 DEBUG:boto:Token: None 2024-03-04T16:18:18.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.388 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3af0d766494c0b-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.390 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.390 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.390 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.390 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.391 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.391 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.391 DEBUG:boto:Data: 2024-03-04T16:18:18.391 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.391 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.391 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.391 DEBUG:boto:Params: {} 2024-03-04T16:18:18.391 DEBUG:boto:Token: None 2024-03-04T16:18:18.391 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw= 2024-03-04T16:18:18.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw=', 'Content-Length': '0'} 2024-03-04T16:18:18.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddd1af796b6b59a3-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.393 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.394 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.394 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.394 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.394 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.394 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.394 DEBUG:boto:Data: 2024-03-04T16:18:18.394 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.394 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.394 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.394 DEBUG:boto:Params: {} 2024-03-04T16:18:18.394 DEBUG:boto:Token: None 2024-03-04T16:18:18.395 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw= 2024-03-04T16:18:18.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw=', 'Content-Length': '0'} 2024-03-04T16:18:18.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc21f4f2c22bf568-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.397 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.397 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.397 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.397 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.397 DEBUG:boto:Method: GET 2024-03-04T16:18:18.397 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.397 DEBUG:boto:Data: 2024-03-04T16:18:18.397 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.398 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.398 DEBUG:boto:Params: {} 2024-03-04T16:18:18.398 DEBUG:boto:Token: None 2024-03-04T16:18:18.398 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e16f2b880d6ce6af-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.400 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.400 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.400 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.400 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.400 DEBUG:boto:Method: GET 2024-03-04T16:18:18.400 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.401 DEBUG:boto:Data: 2024-03-04T16:18:18.401 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.401 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.401 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.401 DEBUG:boto:Params: {} 2024-03-04T16:18:18.401 DEBUG:boto:Token: None 2024-03-04T16:18:18.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d760bac5418c7b00-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.403 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.403 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.403 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.404 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.404 DEBUG:boto:Method: GET 2024-03-04T16:18:18.404 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.404 DEBUG:boto:Data: 2024-03-04T16:18:18.404 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.404 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.404 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.404 DEBUG:boto:Params: {} 2024-03-04T16:18:18.404 DEBUG:boto:Token: None 2024-03-04T16:18:18.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc26b61c7264ac0b-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.407 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.407 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.407 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.407 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.407 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.407 DEBUG:boto:Method: GET 2024-03-04T16:18:18.408 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.408 DEBUG:boto:Data: 2024-03-04T16:18:18.408 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.408 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.408 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.408 DEBUG:boto:Params: {} 2024-03-04T16:18:18.408 DEBUG:boto:Token: None 2024-03-04T16:18:18.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.408 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab1cadf323873eb-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.415 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.415 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.416 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.416 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.416 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.416 DEBUG:boto:Data: 2024-03-04T16:18:18.416 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.416 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.416 DEBUG:boto:Params: {} 2024-03-04T16:18:18.416 DEBUG:boto:Token: None 2024-03-04T16:18:18.416 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg= 2024-03-04T16:18:18.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg=', 'Content-Length': '0'} 2024-03-04T16:18:18.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000668a1acbcfa392be-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.419 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.419 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.419 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.419 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.419 DEBUG:boto:Data: 2024-03-04T16:18:18.419 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.419 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.419 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.419 DEBUG:boto:Params: {} 2024-03-04T16:18:18.419 DEBUG:boto:Token: None 2024-03-04T16:18:18.420 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.420 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg= 2024-03-04T16:18:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg=', 'Content-Length': '0'} 2024-03-04T16:18:18.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738d6c6f45602580-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.422 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.422 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.422 DEBUG:boto:Method: GET 2024-03-04T16:18:18.422 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.422 DEBUG:boto:Data: 2024-03-04T16:18:18.422 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.423 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.423 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.423 DEBUG:boto:Params: {} 2024-03-04T16:18:18.423 DEBUG:boto:Token: None 2024-03-04T16:18:18.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.423 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E= 2024-03-04T16:18:18.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E=', 'Content-Length': '0'} 2024-03-04T16:18:18.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085057a70413f085c-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.468 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.468 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.468 DEBUG:boto:Method: GET 2024-03-04T16:18:18.468 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.468 DEBUG:boto:Data: 2024-03-04T16:18:18.468 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.469 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.469 DEBUG:boto:Params: {} 2024-03-04T16:18:18.469 DEBUG:boto:Token: None 2024-03-04T16:18:18.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.469 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E= 2024-03-04T16:18:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E=', 'Content-Length': '0'} 2024-03-04T16:18:18.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003072620f893efbf8-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.471 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.471 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.471 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.471 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.471 DEBUG:boto:Method: GET 2024-03-04T16:18:18.471 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.471 DEBUG:boto:Data: 2024-03-04T16:18:18.471 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.472 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.472 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.472 DEBUG:boto:Params: {} 2024-03-04T16:18:18.472 DEBUG:boto:Token: None 2024-03-04T16:18:18.472 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.472 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000865d2b8c8439d17a-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.516 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.516 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.517 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.517 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.517 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.517 DEBUG:boto:Method: GET 2024-03-04T16:18:18.517 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.517 DEBUG:boto:Data: 2024-03-04T16:18:18.517 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.517 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.517 DEBUG:boto:Params: {} 2024-03-04T16:18:18.517 DEBUG:boto:Token: None 2024-03-04T16:18:18.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe1cac9f20a35b65-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.520 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-15 zones={test-zone1, test-zone3} 2024-03-04T16:18:18.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.521 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:18.581 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:18.581 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:18.633 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:18.633+0000 7f3c24b2d800 1 bucket sync caught up with source: 2024-03-04T16:18:18.633 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:18.633 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:18.633 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:18.633+0000 7f3c24b2d800 0 bucket checkpoint complete 2024-03-04T16:18:18.653 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-15 zones={test-zone2, test-zone3} 2024-03-04T16:18:18.653 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.653 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.653 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.653 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:18:18.653 DEBUG:boto:Data: 2024-03-04T16:18:18.654 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.654 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.654 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.654 DEBUG:boto:Params: {} 2024-03-04T16:18:18.654 DEBUG:boto:Token: None 2024-03-04T16:18:18.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw= 2024-03-04T16:18:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw=', 'Content-Length': '0'} 2024-03-04T16:18:18.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8fa83d389e21027-0065e5f44a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.657 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.657 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.657 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.657 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:18:18.657 DEBUG:boto:Data: 2024-03-04T16:18:18.657 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.657 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.657 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.657 DEBUG:boto:Params: {} 2024-03-04T16:18:18.658 DEBUG:boto:Token: None 2024-03-04T16:18:18.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.658 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw= 2024-03-04T16:18:18.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tc+t9ybj61IaLKCN9SrTnfC4/vw=', 'Content-Length': '0'} 2024-03-04T16:18:18.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bfae3a95906e007-0065e5f44a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:18.660 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.660 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.660 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.660 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.660 DEBUG:boto:Method: GET 2024-03-04T16:18:18.661 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.661 DEBUG:boto:Data: 2024-03-04T16:18:18.661 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.661 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.661 DEBUG:boto:Params: {} 2024-03-04T16:18:18.661 DEBUG:boto:Token: None 2024-03-04T16:18:18.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.661 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840f9279a476d69f-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.704 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.704 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.705 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:18.705 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.705 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.705 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.705 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.705 DEBUG:boto:Method: GET 2024-03-04T16:18:18.705 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.705 DEBUG:boto:Data: 2024-03-04T16:18:18.705 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.705 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.705 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.706 DEBUG:boto:Params: {} 2024-03-04T16:18:18.706 DEBUG:boto:Token: None 2024-03-04T16:18:18.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0cc939a4aad7cb3-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.709 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.709 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-03-04T16:18:18.709 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.709 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.709 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.709 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.709 DEBUG:boto:Method: GET 2024-03-04T16:18:18.710 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.710 DEBUG:boto:Data: 2024-03-04T16:18:18.710 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.710 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.710 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.710 DEBUG:boto:Params: {} 2024-03-04T16:18:18.710 DEBUG:boto:Token: None 2024-03-04T16:18:18.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf96c6c7898b175-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.713 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.713 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.713 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.713 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.713 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.714 DEBUG:boto:Method: GET 2024-03-04T16:18:18.714 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:18:18.714 DEBUG:boto:Data: 2024-03-04T16:18:18.714 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.714 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.714 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.714 DEBUG:boto:Params: {} 2024-03-04T16:18:18.714 DEBUG:boto:Token: None 2024-03-04T16:18:18.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/?versions 2024-03-04T16:18:18.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE= 2024-03-04T16:18:18.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mYe6T+LX824riCuORGV4NKeawEE=', 'Content-Length': '0'} 2024-03-04T16:18:18.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c5ff5c24e7ff07a-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.717 DEBUG:boto:b'sbnvju-15999falsemyobjnulltrue2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.717 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.717 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.717 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.717 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.718 DEBUG:boto:Method: GET 2024-03-04T16:18:18.718 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.718 DEBUG:boto:Data: 2024-03-04T16:18:18.718 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.718 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.718 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.718 DEBUG:boto:Params: {} 2024-03-04T16:18:18.718 DEBUG:boto:Token: None 2024-03-04T16:18:18.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7a9f3f2cef94af2-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.720 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.720 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.721 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.721 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.721 DEBUG:boto:Method: GET 2024-03-04T16:18:18.721 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.721 DEBUG:boto:Data: 2024-03-04T16:18:18.721 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.721 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.721 DEBUG:boto:Params: {} 2024-03-04T16:18:18.721 DEBUG:boto:Token: None 2024-03-04T16:18:18.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083141dd4af16aeeb-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.764 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.764 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.764 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.764 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.765 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.765 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.765 DEBUG:boto:Data: 2024-03-04T16:18:18.765 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.765 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.765 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.765 DEBUG:boto:Params: {} 2024-03-04T16:18:18.765 DEBUG:boto:Token: None 2024-03-04T16:18:18.766 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw= 2024-03-04T16:18:18.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw=', 'Content-Length': '0'} 2024-03-04T16:18:18.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000502af59d281c328-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.768 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.768 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.768 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.768 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.768 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.768 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.768 DEBUG:boto:Data: 2024-03-04T16:18:18.768 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.769 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.769 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.769 DEBUG:boto:Params: {} 2024-03-04T16:18:18.769 DEBUG:boto:Token: None 2024-03-04T16:18:18.769 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.769 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw= 2024-03-04T16:18:18.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZOjwrCjhxM7YH3a347BR2NP9Hgw=', 'Content-Length': '0'} 2024-03-04T16:18:18.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9aafdd1eb534476-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.771 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.771 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.771 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.771 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.771 DEBUG:boto:Method: GET 2024-03-04T16:18:18.772 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.772 DEBUG:boto:Data: 2024-03-04T16:18:18.772 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.772 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.772 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.772 DEBUG:boto:Params: {} 2024-03-04T16:18:18.772 DEBUG:boto:Token: None 2024-03-04T16:18:18.772 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.772 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de31cf93b7a2312f-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.774 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.775 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.775 DEBUG:boto:path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.775 DEBUG:boto:auth_path=/sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.775 DEBUG:boto:Method: GET 2024-03-04T16:18:18.775 DEBUG:boto:Path: /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.775 DEBUG:boto:Data: 2024-03-04T16:18:18.775 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.775 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.775 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.775 DEBUG:boto:Params: {} 2024-03-04T16:18:18.775 DEBUG:boto:Token: None 2024-03-04T16:18:18.775 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj?versionId=null 2024-03-04T16:18:18.776 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U= 2024-03-04T16:18:18.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:trbVSMfQ+YqvrKb5FjOcx+IAD8U=', 'Content-Length': '0'} 2024-03-04T16:18:18.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fc95a5e3972dd12-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.777 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.777 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.778 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.778 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.778 DEBUG:boto:Method: GET 2024-03-04T16:18:18.778 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.778 DEBUG:boto:Data: 2024-03-04T16:18:18.778 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.778 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.778 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.778 DEBUG:boto:Params: {} 2024-03-04T16:18:18.778 DEBUG:boto:Token: None 2024-03-04T16:18:18.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d52d6bb3dd41a92f-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.781 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.781 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.781 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.781 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.781 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.781 DEBUG:boto:Method: GET 2024-03-04T16:18:18.782 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.782 DEBUG:boto:Data: 2024-03-04T16:18:18.782 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.782 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.782 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.782 DEBUG:boto:Params: {} 2024-03-04T16:18:18.782 DEBUG:boto:Token: None 2024-03-04T16:18:18.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f90ba648f39ad5a-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.785 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.785 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.785 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.785 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.785 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.786 DEBUG:boto:Data: 2024-03-04T16:18:18.786 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.786 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.786 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.786 DEBUG:boto:Params: {} 2024-03-04T16:18:18.786 DEBUG:boto:Token: None 2024-03-04T16:18:18.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg= 2024-03-04T16:18:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg=', 'Content-Length': '0'} 2024-03-04T16:18:18.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043cf8118490be516-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.788 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.788 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.788 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.788 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.788 DEBUG:boto:Data: 2024-03-04T16:18:18.789 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.789 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.789 DEBUG:boto:Params: {} 2024-03-04T16:18:18.789 DEBUG:boto:Token: None 2024-03-04T16:18:18.789 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg= 2024-03-04T16:18:18.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:amviVzdM7co64iBqzs/NUH8HUSg=', 'Content-Length': '0'} 2024-03-04T16:18:18.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000603832d81244e560-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-03-04T16:18:18.791 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.791 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.791 DEBUG:boto:Method: GET 2024-03-04T16:18:18.791 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.791 DEBUG:boto:Data: 2024-03-04T16:18:18.791 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.791 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.792 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.792 DEBUG:boto:Params: {} 2024-03-04T16:18:18.792 DEBUG:boto:Token: None 2024-03-04T16:18:18.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E= 2024-03-04T16:18:18.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E=', 'Content-Length': '0'} 2024-03-04T16:18:18.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-15:fc294220-5038-4f25-aeee-cac2c5838729.4341.24'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d3fae456abd3a6-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.836 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:18.836 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:18.836 DEBUG:boto:Method: GET 2024-03-04T16:18:18.836 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:18.837 DEBUG:boto:Data: 2024-03-04T16:18:18.837 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.837 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.837 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.837 DEBUG:boto:Params: {} 2024-03-04T16:18:18.837 DEBUG:boto:Token: None 2024-03-04T16:18:18.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/myobj 2024-03-04T16:18:18.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E= 2024-03-04T16:18:18.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4rMjbNL5HSYWvEw/2drdjl2kk1E=', 'Content-Length': '0'} 2024-03-04T16:18:18.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0fcedac76be6a09-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.839 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.839 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.839 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.840 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.840 DEBUG:boto:Method: GET 2024-03-04T16:18:18.840 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.840 DEBUG:boto:Data: 2024-03-04T16:18:18.840 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.840 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:18.840 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.840 DEBUG:boto:Params: {} 2024-03-04T16:18:18.840 DEBUG:boto:Token: None 2024-03-04T16:18:18.840 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.841 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bd546732eb5e346-0065e5f44a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.843 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.843 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:18.843 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:18.843 DEBUG:boto:path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.843 DEBUG:boto:auth_path=/sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.844 DEBUG:boto:Method: GET 2024-03-04T16:18:18.844 DEBUG:boto:Path: /sbnvju-15/?prefix=myobj 2024-03-04T16:18:18.844 DEBUG:boto:Data: 2024-03-04T16:18:18.844 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.844 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:18.844 DEBUG:boto:Port: 8001 2024-03-04T16:18:18.844 DEBUG:boto:Params: {} 2024-03-04T16:18:18.844 DEBUG:boto:Token: None 2024-03-04T16:18:18.844 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-15/ 2024-03-04T16:18:18.844 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w= 2024-03-04T16:18:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RPkF2pGQ+JiVZ8Kbj/NC7Kt+6w=', 'Content-Length': '0'} 2024-03-04T16:18:18.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd62d8b59383f23-0065e5f44a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.847 DEBUG:boto:b'sbnvju-15myobj1000falsemyobj2024-03-04T16:17:01.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:18:18.847 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-15 zones={test-zone2, test-zone3} 2024-03-04T16:18:18.847 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:18.847 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:18.847 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.848 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:18:18.848 DEBUG:boto:Data: 2024-03-04T16:18:18.848 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.848 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.848 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.848 DEBUG:boto:Params: {} 2024-03-04T16:18:18.848 DEBUG:boto:Token: None 2024-03-04T16:18:18.848 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-13/ 2024-03-04T16:18:18.848 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWEp6G1xKRRn6OUuk2pI9wNzZBY= 2024-03-04T16:18:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWEp6G1xKRRn6OUuk2pI9wNzZBY=', 'Content-Length': '0'} 2024-03-04T16:18:18.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5962177f0896fd9-0065e5f44a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.851 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:18:18.851 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:18:18.851 DEBUG:boto:Method: HEAD 2024-03-04T16:18:18.851 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:18:18.851 DEBUG:boto:Data: 2024-03-04T16:18:18.851 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.851 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.851 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.852 DEBUG:boto:Params: {} 2024-03-04T16:18:18.852 DEBUG:boto:Token: None 2024-03-04T16:18:18.852 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-13/myobj 2024-03-04T16:18:18.852 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uozlf4s6B8NsZactxBlkL3mRbUA= 2024-03-04T16:18:18.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uozlf4s6B8NsZactxBlkL3mRbUA=', 'Content-Length': '0'} 2024-03-04T16:18:18.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6106037c1fbe440-0065e5f44a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.854 DEBUG:boto:path=/sbnvju-13/myobj 2024-03-04T16:18:18.854 DEBUG:boto:auth_path=/sbnvju-13/myobj 2024-03-04T16:18:18.854 DEBUG:boto:Method: DELETE 2024-03-04T16:18:18.854 DEBUG:boto:Path: /sbnvju-13/myobj 2024-03-04T16:18:18.854 DEBUG:boto:Data: 2024-03-04T16:18:18.854 DEBUG:boto:Headers: {} 2024-03-04T16:18:18.854 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:18.854 DEBUG:boto:Port: 8000 2024-03-04T16:18:18.854 DEBUG:boto:Params: {} 2024-03-04T16:18:18.854 DEBUG:boto:Token: None 2024-03-04T16:18:18.855 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:18:18 GMT /sbnvju-13/myobj 2024-03-04T16:18:18.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWWotsElHUTLskIDJwbxzADIDLQ= 2024-03-04T16:18:18.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWWotsElHUTLskIDJwbxzADIDLQ=', 'Content-Length': '0'} 2024-03-04T16:18:18.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3449b360d44d864-0065e5f44a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:18.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:18.862 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:18.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:18.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:18.964+0000 7f795512d800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:18.964 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:18:18.964 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:23.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:23.965+0000 7f795512d800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:23.966 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:18:23.966 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:28.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:28.967+0000 7f795512d800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:28.967 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-03-04T16:18:28.967 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:33.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:33.968+0000 7f795512d800 1 bucket sync caught up with source: 2024-03-04T16:18:33.969 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:33.969 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:33.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:18:33.968+0000 7f795512d800 0 bucket checkpoint complete 2024-03-04T16:18:33.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-13 zones={test-zone2, test-zone1} 2024-03-04T16:18:33.991 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:33.991 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:33.992 DEBUG:boto:Method: HEAD 2024-03-04T16:18:33.992 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:18:33.992 DEBUG:boto:Data: 2024-03-04T16:18:33.992 DEBUG:boto:Headers: {} 2024-03-04T16:18:33.992 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:33.992 DEBUG:boto:Port: 8000 2024-03-04T16:18:33.992 DEBUG:boto:Params: {} 2024-03-04T16:18:33.992 DEBUG:boto:Token: None 2024-03-04T16:18:33.992 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:33 GMT /sbnvju-13/ 2024-03-04T16:18:33.993 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZNkRr/ppThTkJfCpaqTW5xLZquE= 2024-03-04T16:18:33.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:33 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZNkRr/ppThTkJfCpaqTW5xLZquE=', 'Content-Length': '0'} 2024-03-04T16:18:33.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f5a45c961cefc9d-0065e5f459-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:33.996 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:33.996 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:33.996 DEBUG:boto:Method: HEAD 2024-03-04T16:18:33.996 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:18:33.996 DEBUG:boto:Data: 2024-03-04T16:18:33.996 DEBUG:boto:Headers: {} 2024-03-04T16:18:33.996 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:33.996 DEBUG:boto:Port: 8000 2024-03-04T16:18:33.996 DEBUG:boto:Params: {} 2024-03-04T16:18:33.996 DEBUG:boto:Token: None 2024-03-04T16:18:33.997 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:33 GMT /sbnvju-13/ 2024-03-04T16:18:33.997 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZNkRr/ppThTkJfCpaqTW5xLZquE= 2024-03-04T16:18:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:33 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZNkRr/ppThTkJfCpaqTW5xLZquE=', 'Content-Length': '0'} 2024-03-04T16:18:33.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0c71647d9f5b945-0065e5f459-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:33 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:33.999 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:34.000 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.000 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.000 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.000 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.000 DEBUG:boto:Method: GET 2024-03-04T16:18:34.000 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.000 DEBUG:boto:Data: 2024-03-04T16:18:34.000 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.000 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:34.000 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.001 DEBUG:boto:Params: {} 2024-03-04T16:18:34.001 DEBUG:boto:Token: None 2024-03-04T16:18:34.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19a0b6c2a625494-0065e5f45a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.003 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:34.004 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.004 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.004 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.004 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.004 DEBUG:boto:Method: GET 2024-03-04T16:18:34.004 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.004 DEBUG:boto:Data: 2024-03-04T16:18:34.005 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.005 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:34.005 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.005 DEBUG:boto:Params: {} 2024-03-04T16:18:34.005 DEBUG:boto:Token: None 2024-03-04T16:18:34.005 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.005 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000efac5cba7b4a056-0065e5f45a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.008 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.008 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.008 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.008 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.008 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.008 DEBUG:boto:Method: GET 2024-03-04T16:18:34.009 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.009 DEBUG:boto:Data: 2024-03-04T16:18:34.009 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:34.009 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.009 DEBUG:boto:Params: {} 2024-03-04T16:18:34.009 DEBUG:boto:Token: None 2024-03-04T16:18:34.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.009 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a119145266afb64-0065e5f45a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.012 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.012 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.012 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.012 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.012 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.012 DEBUG:boto:Method: GET 2024-03-04T16:18:34.012 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.012 DEBUG:boto:Data: 2024-03-04T16:18:34.012 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:34.013 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.013 DEBUG:boto:Params: {} 2024-03-04T16:18:34.013 DEBUG:boto:Token: None 2024-03-04T16:18:34.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c289118c5fbd5e3-0065e5f45a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.016 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.016 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-13 zones={test-zone2, test-zone1} 2024-03-04T16:18:34.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:34.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:34.017 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:34.132+0000 7fdda8333800 1 bucket sync caught up with source: 2024-03-04T16:18:34.132 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:34.132 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.20.6, , , , , , , , , ] 2024-03-04T16:18:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:34.132+0000 7fdda8333800 0 bucket checkpoint complete 2024-03-04T16:18:34.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-13 zones={test-zone3, test-zone1} 2024-03-04T16:18:34.152 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.152 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.152 DEBUG:boto:Method: HEAD 2024-03-04T16:18:34.152 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:18:34.152 DEBUG:boto:Data: 2024-03-04T16:18:34.152 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.152 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:34.152 DEBUG:boto:Port: 8001 2024-03-04T16:18:34.152 DEBUG:boto:Params: {} 2024-03-04T16:18:34.152 DEBUG:boto:Token: None 2024-03-04T16:18:34.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/ 2024-03-04T16:18:34.153 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w6IoYi7PRPwJH4NU1GWnXwjYvkU= 2024-03-04T16:18:34.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w6IoYi7PRPwJH4NU1GWnXwjYvkU=', 'Content-Length': '0'} 2024-03-04T16:18:34.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa9c2f47bd59fd5b-0065e5f45a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.155 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.156 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.156 DEBUG:boto:Method: HEAD 2024-03-04T16:18:34.156 DEBUG:boto:Path: /sbnvju-13/ 2024-03-04T16:18:34.156 DEBUG:boto:Data: 2024-03-04T16:18:34.156 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.156 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:34.156 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.156 DEBUG:boto:Params: {} 2024-03-04T16:18:34.156 DEBUG:boto:Token: None 2024-03-04T16:18:34.156 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/ 2024-03-04T16:18:34.156 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w6IoYi7PRPwJH4NU1GWnXwjYvkU= 2024-03-04T16:18:34.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w6IoYi7PRPwJH4NU1GWnXwjYvkU=', 'Content-Length': '0'} 2024-03-04T16:18:34.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d7306dcffc3c4cc-0065e5f45a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:34.159 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.159 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.159 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.159 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.159 DEBUG:boto:Method: GET 2024-03-04T16:18:34.159 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.160 DEBUG:boto:Data: 2024-03-04T16:18:34.160 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.160 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:34.160 DEBUG:boto:Port: 8001 2024-03-04T16:18:34.160 DEBUG:boto:Params: {} 2024-03-04T16:18:34.160 DEBUG:boto:Token: None 2024-03-04T16:18:34.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.160 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b91fe030d68f64-0065e5f45a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.163 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.163 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:34.163 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.163 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.163 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.164 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.164 DEBUG:boto:Method: GET 2024-03-04T16:18:34.164 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.164 DEBUG:boto:Data: 2024-03-04T16:18:34.164 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.164 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:34.164 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.164 DEBUG:boto:Params: {} 2024-03-04T16:18:34.164 DEBUG:boto:Token: None 2024-03-04T16:18:34.165 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c651087ccac6c07-0065e5f45a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.167 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.167 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.167 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.168 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.168 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.168 DEBUG:boto:Method: GET 2024-03-04T16:18:34.168 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.168 DEBUG:boto:Data: 2024-03-04T16:18:34.168 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.168 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:34.168 DEBUG:boto:Port: 8001 2024-03-04T16:18:34.168 DEBUG:boto:Params: {} 2024-03-04T16:18:34.168 DEBUG:boto:Token: None 2024-03-04T16:18:34.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16b301fc29f5e7d-0065e5f45a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.171 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.171 DEBUG:boto:path=/sbnvju-13/ 2024-03-04T16:18:34.171 DEBUG:boto:auth_path=/sbnvju-13/ 2024-03-04T16:18:34.171 DEBUG:boto:path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.171 DEBUG:boto:auth_path=/sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.171 DEBUG:boto:Method: GET 2024-03-04T16:18:34.171 DEBUG:boto:Path: /sbnvju-13/?versions&max-keys=999 2024-03-04T16:18:34.172 DEBUG:boto:Data: 2024-03-04T16:18:34.172 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.172 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:34.172 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.172 DEBUG:boto:Params: {} 2024-03-04T16:18:34.172 DEBUG:boto:Token: None 2024-03-04T16:18:34.172 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-13/?versions 2024-03-04T16:18:34.172 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU= 2024-03-04T16:18:34.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IICn/AyJeJPpBJs1wcmnLZAA6xU=', 'Content-Length': '0'} 2024-03-04T16:18:34.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fafb8fb3474437f7-0065e5f45a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.174 DEBUG:boto:b'sbnvju-13999false' 2024-03-04T16:18:34.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-13 zones={test-zone3, test-zone1} 2024-03-04T16:18:34.175 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:34.175 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:34.175 DEBUG:boto:Method: HEAD 2024-03-04T16:18:34.175 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:34.175 DEBUG:boto:Data: 2024-03-04T16:18:34.175 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.175 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:34.175 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.175 DEBUG:boto:Params: {} 2024-03-04T16:18:34.176 DEBUG:boto:Token: None 2024-03-04T16:18:34.176 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-14/ 2024-03-04T16:18:34.176 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iw3qOHSUB9+j8885ypydN6irq3U= 2024-03-04T16:18:34.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iw3qOHSUB9+j8885ypydN6irq3U=', 'Content-Length': '0'} 2024-03-04T16:18:34.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa6db9a169be726-0065e5f45a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.178 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:34.179 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:34.179 DEBUG:boto:Method: HEAD 2024-03-04T16:18:34.179 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:34.179 DEBUG:boto:Data: 2024-03-04T16:18:34.179 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.179 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:34.179 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.179 DEBUG:boto:Params: {} 2024-03-04T16:18:34.179 DEBUG:boto:Token: None 2024-03-04T16:18:34.179 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-14/myobj 2024-03-04T16:18:34.180 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Go6u9NFcC1OqxQeYydnri8fKzsE= 2024-03-04T16:18:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Go6u9NFcC1OqxQeYydnri8fKzsE=', 'Content-Length': '0'} 2024-03-04T16:18:34.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f30502404f9d738e-0065e5f45a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.182 DEBUG:boto:path=/sbnvju-14/myobj 2024-03-04T16:18:34.182 DEBUG:boto:auth_path=/sbnvju-14/myobj 2024-03-04T16:18:34.182 DEBUG:boto:Method: DELETE 2024-03-04T16:18:34.182 DEBUG:boto:Path: /sbnvju-14/myobj 2024-03-04T16:18:34.182 DEBUG:boto:Data: 2024-03-04T16:18:34.182 DEBUG:boto:Headers: {} 2024-03-04T16:18:34.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:34.182 DEBUG:boto:Port: 8000 2024-03-04T16:18:34.182 DEBUG:boto:Params: {} 2024-03-04T16:18:34.182 DEBUG:boto:Token: None 2024-03-04T16:18:34.183 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:18:34 GMT /sbnvju-14/myobj 2024-03-04T16:18:34.183 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FCksDH2B1USnIjX1+03bke17B2A= 2024-03-04T16:18:34.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:34 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FCksDH2B1USnIjX1+03bke17B2A=', 'Content-Length': '0'} 2024-03-04T16:18:34.189 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000daa12cd22c8a1d18-0065e5f45a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:34 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:34.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:34.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:34.190 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:34.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:34.297+0000 7f6b2ac0f800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:34.297 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:34.298 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:39.298+0000 7f6b2ac0f800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:39.299 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:39.299 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:44.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:44.300+0000 7f6b2ac0f800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:44.301 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:44.301 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:49.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:49.301+0000 7f6b2ac0f800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:49.302 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-03-04T16:18:49.302 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:54.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:54.303+0000 7f6b2ac0f800 1 bucket sync caught up with source: 2024-03-04T16:18:54.303 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:54.303 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:54.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:54.303+0000 7f6b2ac0f800 0 bucket checkpoint complete 2024-03-04T16:18:54.328 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-14 zones={test-zone1, test-zone2} 2024-03-04T16:18:54.328 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.328 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.328 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.329 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:54.329 DEBUG:boto:Data: 2024-03-04T16:18:54.329 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:54.329 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.329 DEBUG:boto:Params: {} 2024-03-04T16:18:54.329 DEBUG:boto:Token: None 2024-03-04T16:18:54.330 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/ 2024-03-04T16:18:54.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0= 2024-03-04T16:18:54.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0=', 'Content-Length': '0'} 2024-03-04T16:18:54.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a491829587e4f5b5-0065e5f46e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.333 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.334 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.334 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.334 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:54.334 DEBUG:boto:Data: 2024-03-04T16:18:54.334 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.334 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.334 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.334 DEBUG:boto:Params: {} 2024-03-04T16:18:54.334 DEBUG:boto:Token: None 2024-03-04T16:18:54.334 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/ 2024-03-04T16:18:54.334 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0= 2024-03-04T16:18:54.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0=', 'Content-Length': '0'} 2024-03-04T16:18:54.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc1f05fac10b004d-0065e5f46e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:54.338 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.338 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.338 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.338 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.338 DEBUG:boto:Method: GET 2024-03-04T16:18:54.338 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.338 DEBUG:boto:Data: 2024-03-04T16:18:54.338 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.338 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:54.338 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.338 DEBUG:boto:Params: {} 2024-03-04T16:18:54.339 DEBUG:boto:Token: None 2024-03-04T16:18:54.339 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.339 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000914c16886f88bd89-0065e5f46e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.342 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:54.343 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.343 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.343 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.343 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.343 DEBUG:boto:Method: GET 2024-03-04T16:18:54.343 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.343 DEBUG:boto:Data: 2024-03-04T16:18:54.343 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.343 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.343 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.343 DEBUG:boto:Params: {} 2024-03-04T16:18:54.343 DEBUG:boto:Token: None 2024-03-04T16:18:54.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.344 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017977d95d70e6935-0065e5f46e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.347 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.347 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.347 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.347 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.347 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.347 DEBUG:boto:Method: GET 2024-03-04T16:18:54.347 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.347 DEBUG:boto:Data: 2024-03-04T16:18:54.347 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.347 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:18:54.348 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.348 DEBUG:boto:Params: {} 2024-03-04T16:18:54.348 DEBUG:boto:Token: None 2024-03-04T16:18:54.348 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.348 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c200c33778212c07-0065e5f46e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.351 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.351 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.351 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.351 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.351 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.351 DEBUG:boto:Method: GET 2024-03-04T16:18:54.351 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.351 DEBUG:boto:Data: 2024-03-04T16:18:54.352 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.352 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.352 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.352 DEBUG:boto:Params: {} 2024-03-04T16:18:54.352 DEBUG:boto:Token: None 2024-03-04T16:18:54.352 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.352 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c5a982dce104880-0065e5f46e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.355 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.355 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-14 zones={test-zone1, test-zone2} 2024-03-04T16:18:54.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:54.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:54.356 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:54.415 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:54.415 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:54.471+0000 7f7e22d30800 1 bucket sync caught up with source: 2024-03-04T16:18:54.471 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:54.471 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.23.6, , , , , , , , , ] 2024-03-04T16:18:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:54.471+0000 7f7e22d30800 0 bucket checkpoint complete 2024-03-04T16:18:54.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-14 zones={test-zone3, test-zone2} 2024-03-04T16:18:54.490 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.490 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.490 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.490 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:54.490 DEBUG:boto:Data: 2024-03-04T16:18:54.490 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.490 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.491 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.491 DEBUG:boto:Params: {} 2024-03-04T16:18:54.491 DEBUG:boto:Token: None 2024-03-04T16:18:54.491 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/ 2024-03-04T16:18:54.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0= 2024-03-04T16:18:54.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0=', 'Content-Length': '0'} 2024-03-04T16:18:54.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087b24af1f5e5d6af-0065e5f46e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.494 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.494 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.494 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.494 DEBUG:boto:Path: /sbnvju-14/ 2024-03-04T16:18:54.494 DEBUG:boto:Data: 2024-03-04T16:18:54.494 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.494 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.494 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.494 DEBUG:boto:Params: {} 2024-03-04T16:18:54.494 DEBUG:boto:Token: None 2024-03-04T16:18:54.495 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/ 2024-03-04T16:18:54.495 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0= 2024-03-04T16:18:54.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gW6jsMDR8BKa5g+okVtgCS8eEy0=', 'Content-Length': '0'} 2024-03-04T16:18:54.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f1af393a8714177-0065e5f46e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:18:54.497 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.497 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.498 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.498 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.498 DEBUG:boto:Method: GET 2024-03-04T16:18:54.498 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.498 DEBUG:boto:Data: 2024-03-04T16:18:54.498 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.498 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.498 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.498 DEBUG:boto:Params: {} 2024-03-04T16:18:54.498 DEBUG:boto:Token: None 2024-03-04T16:18:54.498 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.499 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e2260436d45875-0065e5f46e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.544 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.544 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:18:54.544 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.545 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.545 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.545 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.545 DEBUG:boto:Method: GET 2024-03-04T16:18:54.545 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.545 DEBUG:boto:Data: 2024-03-04T16:18:54.545 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.545 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.545 DEBUG:boto:Params: {} 2024-03-04T16:18:54.545 DEBUG:boto:Token: None 2024-03-04T16:18:54.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005375d864edcd9866-0065e5f46e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.548 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.548 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.549 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.549 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.549 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.549 DEBUG:boto:Method: GET 2024-03-04T16:18:54.549 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.549 DEBUG:boto:Data: 2024-03-04T16:18:54.549 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.549 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.549 DEBUG:boto:Params: {} 2024-03-04T16:18:54.550 DEBUG:boto:Token: None 2024-03-04T16:18:54.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca1e9eaa182448b-0065e5f46e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.553 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.553 DEBUG:boto:path=/sbnvju-14/ 2024-03-04T16:18:54.553 DEBUG:boto:auth_path=/sbnvju-14/ 2024-03-04T16:18:54.553 DEBUG:boto:path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.553 DEBUG:boto:auth_path=/sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.553 DEBUG:boto:Method: GET 2024-03-04T16:18:54.553 DEBUG:boto:Path: /sbnvju-14/?versions&max-keys=999 2024-03-04T16:18:54.554 DEBUG:boto:Data: 2024-03-04T16:18:54.554 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.554 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:18:54.554 DEBUG:boto:Port: 8000 2024-03-04T16:18:54.554 DEBUG:boto:Params: {} 2024-03-04T16:18:54.554 DEBUG:boto:Token: None 2024-03-04T16:18:54.554 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-14/?versions 2024-03-04T16:18:54.554 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg= 2024-03-04T16:18:54.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bhKvGvDUK7DUHub1bPUhjXy0SRg=', 'Content-Length': '0'} 2024-03-04T16:18:54.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007387194e87023aef-0065e5f46e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.600 DEBUG:boto:b'sbnvju-14999false' 2024-03-04T16:18:54.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-14 zones={test-zone3, test-zone2} 2024-03-04T16:18:54.600 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:18:54.601 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:18:54.601 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.601 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:18:54.601 DEBUG:boto:Data: 2024-03-04T16:18:54.601 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.601 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.601 DEBUG:boto:Params: {} 2024-03-04T16:18:54.601 DEBUG:boto:Token: None 2024-03-04T16:18:54.601 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-15/ 2024-03-04T16:18:54.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JOnYqwrovDHcYnWi8EP+vNaQ4Ws= 2024-03-04T16:18:54.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JOnYqwrovDHcYnWi8EP+vNaQ4Ws=', 'Content-Length': '0'} 2024-03-04T16:18:54.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000996e5fb654e4772d-0065e5f46e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.604 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:54.604 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:54.604 DEBUG:boto:Method: HEAD 2024-03-04T16:18:54.604 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:54.604 DEBUG:boto:Data: 2024-03-04T16:18:54.604 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.605 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.605 DEBUG:boto:Params: {} 2024-03-04T16:18:54.605 DEBUG:boto:Token: None 2024-03-04T16:18:54.605 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-15/myobj 2024-03-04T16:18:54.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iPSpypJyOsC4VAV/nhFAhoEz4o0= 2024-03-04T16:18:54.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iPSpypJyOsC4VAV/nhFAhoEz4o0=', 'Content-Length': '0'} 2024-03-04T16:18:54.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:17:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000901ff98485ef4d2d-0065e5f46e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.607 DEBUG:boto:path=/sbnvju-15/myobj 2024-03-04T16:18:54.607 DEBUG:boto:auth_path=/sbnvju-15/myobj 2024-03-04T16:18:54.607 DEBUG:boto:Method: DELETE 2024-03-04T16:18:54.607 DEBUG:boto:Path: /sbnvju-15/myobj 2024-03-04T16:18:54.607 DEBUG:boto:Data: 2024-03-04T16:18:54.607 DEBUG:boto:Headers: {} 2024-03-04T16:18:54.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:18:54.607 DEBUG:boto:Port: 8001 2024-03-04T16:18:54.607 DEBUG:boto:Params: {} 2024-03-04T16:18:54.608 DEBUG:boto:Token: None 2024-03-04T16:18:54.608 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:18:54 GMT /sbnvju-15/myobj 2024-03-04T16:18:54.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cXhnZYB06++TjOf38z26aww6zBc= 2024-03-04T16:18:54.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:18:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cXhnZYB06++TjOf38z26aww6zBc=', 'Content-Length': '0'} 2024-03-04T16:18:54.613 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c13cd1c1fb0f951-0065e5f46e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:18:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:18:54.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:54.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:18:54.613 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:18:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:18:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:18:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:54.720+0000 7f1569d24800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:54.720 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:54.720 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:18:59.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:18:59.721+0000 7f1569d24800 1 waiting for incremental sync to catch up: 2024-03-04T16:18:59.722 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:18:59.722 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:04.722+0000 7f1569d24800 1 waiting for incremental sync to catch up: 2024-03-04T16:19:04.723 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:19:04.723 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:09.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:09.723+0000 7f1569d24800 1 waiting for incremental sync to catch up: 2024-03-04T16:19:09.724 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000001.20.6, , , , , , , , , ] 2024-03-04T16:19:09.724 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:14.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:14.725+0000 7f1569d24800 1 bucket sync caught up with source: 2024-03-04T16:19:14.726 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:14.726 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:14.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:14.725+0000 7f1569d24800 0 bucket checkpoint complete 2024-03-04T16:19:14.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-15 zones={test-zone1, test-zone3} 2024-03-04T16:19:14.749 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.749 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.750 DEBUG:boto:Method: HEAD 2024-03-04T16:19:14.750 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:19:14.750 DEBUG:boto:Data: 2024-03-04T16:19:14.750 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.750 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:14.750 DEBUG:boto:Port: 8000 2024-03-04T16:19:14.750 DEBUG:boto:Params: {} 2024-03-04T16:19:14.750 DEBUG:boto:Token: None 2024-03-04T16:19:14.750 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/ 2024-03-04T16:19:14.751 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE= 2024-03-04T16:19:14.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE=', 'Content-Length': '0'} 2024-03-04T16:19:14.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046d35fd8a326cd53-0065e5f482-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.753 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.754 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.754 DEBUG:boto:Method: HEAD 2024-03-04T16:19:14.754 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:19:14.754 DEBUG:boto:Data: 2024-03-04T16:19:14.754 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.754 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:14.754 DEBUG:boto:Port: 8001 2024-03-04T16:19:14.754 DEBUG:boto:Params: {} 2024-03-04T16:19:14.754 DEBUG:boto:Token: None 2024-03-04T16:19:14.754 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/ 2024-03-04T16:19:14.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE= 2024-03-04T16:19:14.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE=', 'Content-Length': '0'} 2024-03-04T16:19:14.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055f17c3aea6178ce-0065e5f482-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.757 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:19:14.757 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.757 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.757 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.757 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.757 DEBUG:boto:Method: GET 2024-03-04T16:19:14.758 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.758 DEBUG:boto:Data: 2024-03-04T16:19:14.758 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.758 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:14.758 DEBUG:boto:Port: 8000 2024-03-04T16:19:14.758 DEBUG:boto:Params: {} 2024-03-04T16:19:14.758 DEBUG:boto:Token: None 2024-03-04T16:19:14.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.758 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b851556ae63b3c2e-0065e5f482-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.808 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:14.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:19:14.809 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.809 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.809 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.809 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.809 DEBUG:boto:Method: GET 2024-03-04T16:19:14.809 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.809 DEBUG:boto:Data: 2024-03-04T16:19:14.809 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:14.809 DEBUG:boto:Port: 8001 2024-03-04T16:19:14.809 DEBUG:boto:Params: {} 2024-03-04T16:19:14.810 DEBUG:boto:Token: None 2024-03-04T16:19:14.810 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.810 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1cc8ef0f2514ae-0065e5f482-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.813 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:14.813 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.813 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.813 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.813 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.813 DEBUG:boto:Method: GET 2024-03-04T16:19:14.813 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.814 DEBUG:boto:Data: 2024-03-04T16:19:14.814 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.814 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:14.814 DEBUG:boto:Port: 8000 2024-03-04T16:19:14.814 DEBUG:boto:Params: {} 2024-03-04T16:19:14.814 DEBUG:boto:Token: None 2024-03-04T16:19:14.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a69223feef0243-0065e5f482-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.817 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:14.817 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.817 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.817 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.818 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.818 DEBUG:boto:Method: GET 2024-03-04T16:19:14.818 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.818 DEBUG:boto:Data: 2024-03-04T16:19:14.818 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.818 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:14.818 DEBUG:boto:Port: 8001 2024-03-04T16:19:14.818 DEBUG:boto:Params: {} 2024-03-04T16:19:14.818 DEBUG:boto:Token: None 2024-03-04T16:19:14.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.819 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a510073d6a590521-0065e5f482-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.821 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:14.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-15 zones={test-zone1, test-zone3} 2024-03-04T16:19:14.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:19:14.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:19:14.822 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:19:14.882 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:19:14.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:19:14.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:14.935+0000 7f1bef024800 1 bucket sync caught up with source: 2024-03-04T16:19:14.935 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:14.935 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2024-03-04T16:19:14.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:14.935+0000 7f1bef024800 0 bucket checkpoint complete 2024-03-04T16:19:14.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-15 zones={test-zone2, test-zone3} 2024-03-04T16:19:14.953 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.953 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.953 DEBUG:boto:Method: HEAD 2024-03-04T16:19:14.953 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:19:14.954 DEBUG:boto:Data: 2024-03-04T16:19:14.954 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.954 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:14.954 DEBUG:boto:Port: 8000 2024-03-04T16:19:14.954 DEBUG:boto:Params: {} 2024-03-04T16:19:14.954 DEBUG:boto:Token: None 2024-03-04T16:19:14.954 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/ 2024-03-04T16:19:14.954 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE= 2024-03-04T16:19:14.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE=', 'Content-Length': '0'} 2024-03-04T16:19:14.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000335cf35ce967a7dd-0065e5f482-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.957 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.957 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.957 DEBUG:boto:Method: HEAD 2024-03-04T16:19:14.957 DEBUG:boto:Path: /sbnvju-15/ 2024-03-04T16:19:14.957 DEBUG:boto:Data: 2024-03-04T16:19:14.957 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:14.958 DEBUG:boto:Port: 8001 2024-03-04T16:19:14.958 DEBUG:boto:Params: {} 2024-03-04T16:19:14.958 DEBUG:boto:Token: None 2024-03-04T16:19:14.958 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/ 2024-03-04T16:19:14.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE= 2024-03-04T16:19:14.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8sucrlCuvL2E4dLNslTXZeIwkrE=', 'Content-Length': '0'} 2024-03-04T16:19:14.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000410727cc75702e43-0065e5f482-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.961 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:19:14.961 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.961 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.961 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.961 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.961 DEBUG:boto:Method: GET 2024-03-04T16:19:14.961 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.961 DEBUG:boto:Data: 2024-03-04T16:19:14.962 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.962 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:14.962 DEBUG:boto:Port: 8000 2024-03-04T16:19:14.962 DEBUG:boto:Params: {} 2024-03-04T16:19:14.962 DEBUG:boto:Token: None 2024-03-04T16:19:14.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d71ecaacf23ad7-0065e5f482-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:14.964 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:14.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:19:14.965 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:14.965 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:14.965 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.965 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.965 DEBUG:boto:Method: GET 2024-03-04T16:19:14.965 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:14.965 DEBUG:boto:Data: 2024-03-04T16:19:14.966 DEBUG:boto:Headers: {} 2024-03-04T16:19:14.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:14.966 DEBUG:boto:Port: 8001 2024-03-04T16:19:14.966 DEBUG:boto:Params: {} 2024-03-04T16:19:14.966 DEBUG:boto:Token: None 2024-03-04T16:19:14.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:14 GMT /sbnvju-15/?versions 2024-03-04T16:19:14.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c= 2024-03-04T16:19:14.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZwpPcfzi+4e6Ca6LhynBIvI92c=', 'Content-Length': '0'} 2024-03-04T16:19:14.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee617deecbddb0e-0065e5f482-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.016 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:15.017 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:15.017 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:15.017 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.017 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.017 DEBUG:boto:Method: GET 2024-03-04T16:19:15.017 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.018 DEBUG:boto:Data: 2024-03-04T16:19:15.018 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.018 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:15.018 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.018 DEBUG:boto:Params: {} 2024-03-04T16:19:15.018 DEBUG:boto:Token: None 2024-03-04T16:19:15.018 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-15/?versions 2024-03-04T16:19:15.018 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Qu+kfBGUiPVLQSiFI7/uWRhPypQ= 2024-03-04T16:19:15.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Qu+kfBGUiPVLQSiFI7/uWRhPypQ=', 'Content-Length': '0'} 2024-03-04T16:19:15.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007595cfada6959181-0065e5f483-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.021 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:15.022 DEBUG:boto:path=/sbnvju-15/ 2024-03-04T16:19:15.022 DEBUG:boto:auth_path=/sbnvju-15/ 2024-03-04T16:19:15.022 DEBUG:boto:path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.022 DEBUG:boto:auth_path=/sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.022 DEBUG:boto:Method: GET 2024-03-04T16:19:15.022 DEBUG:boto:Path: /sbnvju-15/?versions&max-keys=999 2024-03-04T16:19:15.022 DEBUG:boto:Data: 2024-03-04T16:19:15.022 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.022 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:15.022 DEBUG:boto:Port: 8001 2024-03-04T16:19:15.022 DEBUG:boto:Params: {} 2024-03-04T16:19:15.023 DEBUG:boto:Token: None 2024-03-04T16:19:15.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-15/?versions 2024-03-04T16:19:15.023 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Qu+kfBGUiPVLQSiFI7/uWRhPypQ= 2024-03-04T16:19:15.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Qu+kfBGUiPVLQSiFI7/uWRhPypQ=', 'Content-Length': '0'} 2024-03-04T16:19:15.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9c41b6cece1222-0065e5f483-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.026 DEBUG:boto:b'sbnvju-15999false' 2024-03-04T16:19:15.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-15 zones={test-zone2, test-zone3} 2024-03-04T16:19:15.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-03-04T16:19:15.028 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-16 2024-03-04T16:19:15.028 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.028 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.028 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.028 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.028 DEBUG:boto:Data: 2024-03-04T16:19:15.028 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.028 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.029 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.029 DEBUG:boto:Params: {} 2024-03-04T16:19:15.029 DEBUG:boto:Token: None 2024-03-04T16:19:15.029 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.029 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0oYN+vdczNc0Y6wYJlmlLXdgkb8= 2024-03-04T16:19:15.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0oYN+vdczNc0Y6wYJlmlLXdgkb8=', 'Content-Length': '0'} 2024-03-04T16:19:15.048 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e784b7ad565e5e9e-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.048 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-17 2024-03-04T16:19:15.048 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:15.049 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:15.049 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.049 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:15.049 DEBUG:boto:Data: 2024-03-04T16:19:15.049 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.049 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:15.049 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.049 DEBUG:boto:Params: {} 2024-03-04T16:19:15.049 DEBUG:boto:Token: None 2024-03-04T16:19:15.050 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-17/ 2024-03-04T16:19:15.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DsNIxYORd6nUbKtOXM/Jt9ZVC54= 2024-03-04T16:19:15.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DsNIxYORd6nUbKtOXM/Jt9ZVC54=', 'Content-Length': '0'} 2024-03-04T16:19:15.078 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000263042343bd4697f-0065e5f483-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.078 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-18 2024-03-04T16:19:15.079 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:15.079 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:15.079 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.079 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:15.079 DEBUG:boto:Data: 2024-03-04T16:19:15.079 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.079 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:15.079 DEBUG:boto:Port: 8001 2024-03-04T16:19:15.079 DEBUG:boto:Params: {} 2024-03-04T16:19:15.079 DEBUG:boto:Token: None 2024-03-04T16:19:15.080 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-18/ 2024-03-04T16:19:15.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cWHV74iBEmx/6KMszUbjISBq0Is= 2024-03-04T16:19:15.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cWHV74iBEmx/6KMszUbjISBq0Is=', 'Content-Length': '0'} 2024-03-04T16:19:15.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d265bd10e152bcc-0065e5f483-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.111 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.111 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.111 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.111 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.111 DEBUG:boto:Data: 2024-03-04T16:19:15.111 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.111 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.111 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.112 DEBUG:boto:Params: {} 2024-03-04T16:19:15.112 DEBUG:boto:Token: None 2024-03-04T16:19:15.112 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.112 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a9de997402fd24f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.115 DEBUG:boto:path=/sbnvju-16/obj1 2024-03-04T16:19:15.115 DEBUG:boto:auth_path=/sbnvju-16/obj1 2024-03-04T16:19:15.115 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.115 DEBUG:boto:Path: /sbnvju-16/obj1 2024-03-04T16:19:15.115 DEBUG:boto:Data: 2024-03-04T16:19:15.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.116 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.116 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.116 DEBUG:boto:Params: {} 2024-03-04T16:19:15.116 DEBUG:boto:Token: None 2024-03-04T16:19:15.117 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj1 2024-03-04T16:19:15.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vGmXhnpuBX7dagK3jBkQ0DTjy6c= 2024-03-04T16:19:15.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vGmXhnpuBX7dagK3jBkQ0DTjy6c='} 2024-03-04T16:19:15.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000564c29e96b77f91c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.124 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.124 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.124 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.124 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.124 DEBUG:boto:Data: 2024-03-04T16:19:15.124 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.124 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.124 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.125 DEBUG:boto:Params: {} 2024-03-04T16:19:15.125 DEBUG:boto:Token: None 2024-03-04T16:19:15.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d3d8ff0f478aa5-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.127 DEBUG:boto:path=/sbnvju-16/obj2 2024-03-04T16:19:15.128 DEBUG:boto:auth_path=/sbnvju-16/obj2 2024-03-04T16:19:15.128 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.128 DEBUG:boto:Path: /sbnvju-16/obj2 2024-03-04T16:19:15.128 DEBUG:boto:Data: 2024-03-04T16:19:15.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.128 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.128 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.128 DEBUG:boto:Params: {} 2024-03-04T16:19:15.129 DEBUG:boto:Token: None 2024-03-04T16:19:15.129 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj2 2024-03-04T16:19:15.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q8abwkRHB5cYVwnfwWWCkbDgdlg= 2024-03-04T16:19:15.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q8abwkRHB5cYVwnfwWWCkbDgdlg='} 2024-03-04T16:19:15.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005541c1b7cb24853c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.134 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.135 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.135 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.135 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.135 DEBUG:boto:Data: 2024-03-04T16:19:15.135 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.135 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.136 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.136 DEBUG:boto:Params: {} 2024-03-04T16:19:15.136 DEBUG:boto:Token: None 2024-03-04T16:19:15.136 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.136 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000361a23308390b84f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.139 DEBUG:boto:path=/sbnvju-16/obj3 2024-03-04T16:19:15.139 DEBUG:boto:auth_path=/sbnvju-16/obj3 2024-03-04T16:19:15.140 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.140 DEBUG:boto:Path: /sbnvju-16/obj3 2024-03-04T16:19:15.140 DEBUG:boto:Data: 2024-03-04T16:19:15.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.140 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.140 DEBUG:boto:Params: {} 2024-03-04T16:19:15.140 DEBUG:boto:Token: None 2024-03-04T16:19:15.140 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj3 2024-03-04T16:19:15.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EyeD5gR2AIwTXIzvXF/c6ZMzrGQ= 2024-03-04T16:19:15.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EyeD5gR2AIwTXIzvXF/c6ZMzrGQ='} 2024-03-04T16:19:15.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a565684f33b34aad-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.184 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.185 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.185 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.185 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.185 DEBUG:boto:Data: 2024-03-04T16:19:15.185 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.185 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.185 DEBUG:boto:Params: {} 2024-03-04T16:19:15.185 DEBUG:boto:Token: None 2024-03-04T16:19:15.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af18401105cdf8b4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.188 DEBUG:boto:path=/sbnvju-16/obj4 2024-03-04T16:19:15.188 DEBUG:boto:auth_path=/sbnvju-16/obj4 2024-03-04T16:19:15.189 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.189 DEBUG:boto:Path: /sbnvju-16/obj4 2024-03-04T16:19:15.189 DEBUG:boto:Data: 2024-03-04T16:19:15.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.189 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.189 DEBUG:boto:Params: {} 2024-03-04T16:19:15.189 DEBUG:boto:Token: None 2024-03-04T16:19:15.189 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj4 2024-03-04T16:19:15.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Hexti9QDaSeihC00vLgRda23I4= 2024-03-04T16:19:15.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Hexti9QDaSeihC00vLgRda23I4='} 2024-03-04T16:19:15.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a5c39fd30558668-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.195 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.195 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.195 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.195 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.195 DEBUG:boto:Data: 2024-03-04T16:19:15.195 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.195 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.195 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.195 DEBUG:boto:Params: {} 2024-03-04T16:19:15.196 DEBUG:boto:Token: None 2024-03-04T16:19:15.196 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.196 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4e4763a328de9be-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.198 DEBUG:boto:path=/sbnvju-16/obj5 2024-03-04T16:19:15.198 DEBUG:boto:auth_path=/sbnvju-16/obj5 2024-03-04T16:19:15.198 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.198 DEBUG:boto:Path: /sbnvju-16/obj5 2024-03-04T16:19:15.198 DEBUG:boto:Data: 2024-03-04T16:19:15.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.198 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.199 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.199 DEBUG:boto:Params: {} 2024-03-04T16:19:15.199 DEBUG:boto:Token: None 2024-03-04T16:19:15.199 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj5 2024-03-04T16:19:15.199 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:U3smLnTNyN6Cb2QnrK6PXT77Moc= 2024-03-04T16:19:15.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:U3smLnTNyN6Cb2QnrK6PXT77Moc='} 2024-03-04T16:19:15.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b908342aafa712-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.244 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.244 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.244 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.245 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.245 DEBUG:boto:Data: 2024-03-04T16:19:15.245 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.245 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.245 DEBUG:boto:Params: {} 2024-03-04T16:19:15.245 DEBUG:boto:Token: None 2024-03-04T16:19:15.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000362072ec4030f8ae-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.248 DEBUG:boto:path=/sbnvju-16/obj6 2024-03-04T16:19:15.248 DEBUG:boto:auth_path=/sbnvju-16/obj6 2024-03-04T16:19:15.248 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.248 DEBUG:boto:Path: /sbnvju-16/obj6 2024-03-04T16:19:15.248 DEBUG:boto:Data: 2024-03-04T16:19:15.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.248 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.249 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.249 DEBUG:boto:Params: {} 2024-03-04T16:19:15.249 DEBUG:boto:Token: None 2024-03-04T16:19:15.249 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj6 2024-03-04T16:19:15.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LeJ/evngH8CqvNHwvP2Bpu5S/UE= 2024-03-04T16:19:15.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LeJ/evngH8CqvNHwvP2Bpu5S/UE='} 2024-03-04T16:19:15.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f4aeb370206dd4c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.254 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.254 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.254 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.254 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.254 DEBUG:boto:Data: 2024-03-04T16:19:15.255 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.255 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.255 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.255 DEBUG:boto:Params: {} 2024-03-04T16:19:15.255 DEBUG:boto:Token: None 2024-03-04T16:19:15.255 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.255 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4f9b064d5741724-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.259 DEBUG:boto:path=/sbnvju-16/obj7 2024-03-04T16:19:15.259 DEBUG:boto:auth_path=/sbnvju-16/obj7 2024-03-04T16:19:15.259 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.259 DEBUG:boto:Path: /sbnvju-16/obj7 2024-03-04T16:19:15.259 DEBUG:boto:Data: 2024-03-04T16:19:15.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.260 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.260 DEBUG:boto:Params: {} 2024-03-04T16:19:15.260 DEBUG:boto:Token: None 2024-03-04T16:19:15.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj7 2024-03-04T16:19:15.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kkMl6vxLC5DnYIzgztvtJ23gx2c= 2024-03-04T16:19:15.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMl6vxLC5DnYIzgztvtJ23gx2c='} 2024-03-04T16:19:15.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000207c38b974efb517-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.308 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.309 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.309 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.309 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.309 DEBUG:boto:Data: 2024-03-04T16:19:15.309 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.309 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.309 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.309 DEBUG:boto:Params: {} 2024-03-04T16:19:15.310 DEBUG:boto:Token: None 2024-03-04T16:19:15.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0b67daef713a409-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.313 DEBUG:boto:path=/sbnvju-16/obj8 2024-03-04T16:19:15.314 DEBUG:boto:auth_path=/sbnvju-16/obj8 2024-03-04T16:19:15.314 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.314 DEBUG:boto:Path: /sbnvju-16/obj8 2024-03-04T16:19:15.314 DEBUG:boto:Data: 2024-03-04T16:19:15.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.314 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.314 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.314 DEBUG:boto:Params: {} 2024-03-04T16:19:15.314 DEBUG:boto:Token: None 2024-03-04T16:19:15.314 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj8 2024-03-04T16:19:15.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O9cCJ8lB1HiJ/bLDc2bdGePUASM= 2024-03-04T16:19:15.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O9cCJ8lB1HiJ/bLDc2bdGePUASM='} 2024-03-04T16:19:15.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044f102e44226cf37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.318 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.318 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.319 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.319 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.319 DEBUG:boto:Data: 2024-03-04T16:19:15.319 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.319 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.319 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.319 DEBUG:boto:Params: {} 2024-03-04T16:19:15.319 DEBUG:boto:Token: None 2024-03-04T16:19:15.319 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.319 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3a0f8962bbf5263-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.322 DEBUG:boto:path=/sbnvju-16/obj9 2024-03-04T16:19:15.322 DEBUG:boto:auth_path=/sbnvju-16/obj9 2024-03-04T16:19:15.322 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.322 DEBUG:boto:Path: /sbnvju-16/obj9 2024-03-04T16:19:15.322 DEBUG:boto:Data: 2024-03-04T16:19:15.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.322 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.323 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.323 DEBUG:boto:Params: {} 2024-03-04T16:19:15.323 DEBUG:boto:Token: None 2024-03-04T16:19:15.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj9 2024-03-04T16:19:15.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wctQePK4Nyxf8gkir+MV2IdAHUQ= 2024-03-04T16:19:15.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wctQePK4Nyxf8gkir+MV2IdAHUQ='} 2024-03-04T16:19:15.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3e66cb42a3e6e9-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.369 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.369 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.369 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.369 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.369 DEBUG:boto:Data: 2024-03-04T16:19:15.369 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.369 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.369 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.369 DEBUG:boto:Params: {} 2024-03-04T16:19:15.369 DEBUG:boto:Token: None 2024-03-04T16:19:15.370 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007466c633bd23a6ac-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.372 DEBUG:boto:path=/sbnvju-16/obj10 2024-03-04T16:19:15.372 DEBUG:boto:auth_path=/sbnvju-16/obj10 2024-03-04T16:19:15.372 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.372 DEBUG:boto:Path: /sbnvju-16/obj10 2024-03-04T16:19:15.372 DEBUG:boto:Data: 2024-03-04T16:19:15.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.372 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.372 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.373 DEBUG:boto:Params: {} 2024-03-04T16:19:15.373 DEBUG:boto:Token: None 2024-03-04T16:19:15.373 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj10 2024-03-04T16:19:15.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2AXDcf4WXCrmkyOpHaYWg3ALcDs= 2024-03-04T16:19:15.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2AXDcf4WXCrmkyOpHaYWg3ALcDs='} 2024-03-04T16:19:15.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027e74d68e579c794-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.376 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.376 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.377 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.377 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.377 DEBUG:boto:Data: 2024-03-04T16:19:15.377 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.377 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.377 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.377 DEBUG:boto:Params: {} 2024-03-04T16:19:15.377 DEBUG:boto:Token: None 2024-03-04T16:19:15.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.377 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092be47adc9b1245c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.379 DEBUG:boto:path=/sbnvju-16/obj11 2024-03-04T16:19:15.380 DEBUG:boto:auth_path=/sbnvju-16/obj11 2024-03-04T16:19:15.380 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.380 DEBUG:boto:Path: /sbnvju-16/obj11 2024-03-04T16:19:15.380 DEBUG:boto:Data: 2024-03-04T16:19:15.380 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.380 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.380 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.380 DEBUG:boto:Params: {} 2024-03-04T16:19:15.380 DEBUG:boto:Token: None 2024-03-04T16:19:15.380 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj11 2024-03-04T16:19:15.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cBssGGDIO2TbJZoGFJQmTyt5DgU= 2024-03-04T16:19:15.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cBssGGDIO2TbJZoGFJQmTyt5DgU='} 2024-03-04T16:19:15.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004449c6f7fe25b854-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.387 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.387 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.387 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.387 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.387 DEBUG:boto:Data: 2024-03-04T16:19:15.387 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.387 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.387 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.387 DEBUG:boto:Params: {} 2024-03-04T16:19:15.388 DEBUG:boto:Token: None 2024-03-04T16:19:15.388 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.388 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cafc349fbbe12ef-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.390 DEBUG:boto:path=/sbnvju-16/obj12 2024-03-04T16:19:15.390 DEBUG:boto:auth_path=/sbnvju-16/obj12 2024-03-04T16:19:15.390 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.390 DEBUG:boto:Path: /sbnvju-16/obj12 2024-03-04T16:19:15.391 DEBUG:boto:Data: 2024-03-04T16:19:15.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.391 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.391 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.391 DEBUG:boto:Params: {} 2024-03-04T16:19:15.391 DEBUG:boto:Token: None 2024-03-04T16:19:15.391 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj12 2024-03-04T16:19:15.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F1TyeAL0nPAclA+D5W3ZZCGrAJY= 2024-03-04T16:19:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F1TyeAL0nPAclA+D5W3ZZCGrAJY='} 2024-03-04T16:19:15.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081279fc71c39548b-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.441 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.441 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.441 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.441 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.441 DEBUG:boto:Data: 2024-03-04T16:19:15.441 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.441 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.441 DEBUG:boto:Params: {} 2024-03-04T16:19:15.441 DEBUG:boto:Token: None 2024-03-04T16:19:15.442 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8f674b914fb3363-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.444 DEBUG:boto:path=/sbnvju-16/obj13 2024-03-04T16:19:15.444 DEBUG:boto:auth_path=/sbnvju-16/obj13 2024-03-04T16:19:15.445 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.445 DEBUG:boto:Path: /sbnvju-16/obj13 2024-03-04T16:19:15.445 DEBUG:boto:Data: 2024-03-04T16:19:15.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.445 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.445 DEBUG:boto:Params: {} 2024-03-04T16:19:15.445 DEBUG:boto:Token: None 2024-03-04T16:19:15.445 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj13 2024-03-04T16:19:15.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tE6SfcMm5E6uarTeiw6rnhxZCns= 2024-03-04T16:19:15.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tE6SfcMm5E6uarTeiw6rnhxZCns='} 2024-03-04T16:19:15.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000829c0064954ddc10-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.449 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.449 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.450 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.450 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.450 DEBUG:boto:Data: 2024-03-04T16:19:15.450 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.450 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.450 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.450 DEBUG:boto:Params: {} 2024-03-04T16:19:15.450 DEBUG:boto:Token: None 2024-03-04T16:19:15.450 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b107b6b19eba6695-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.453 DEBUG:boto:path=/sbnvju-16/obj14 2024-03-04T16:19:15.453 DEBUG:boto:auth_path=/sbnvju-16/obj14 2024-03-04T16:19:15.453 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.453 DEBUG:boto:Path: /sbnvju-16/obj14 2024-03-04T16:19:15.453 DEBUG:boto:Data: 2024-03-04T16:19:15.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.453 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.453 DEBUG:boto:Params: {} 2024-03-04T16:19:15.453 DEBUG:boto:Token: None 2024-03-04T16:19:15.454 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj14 2024-03-04T16:19:15.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d/dRYOGokRcA1QhHXdlRXExtNXg= 2024-03-04T16:19:15.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d/dRYOGokRcA1QhHXdlRXExtNXg='} 2024-03-04T16:19:15.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6c4544d2be67d81-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.501 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.501 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.501 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.501 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.501 DEBUG:boto:Data: 2024-03-04T16:19:15.501 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.501 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.501 DEBUG:boto:Params: {} 2024-03-04T16:19:15.501 DEBUG:boto:Token: None 2024-03-04T16:19:15.501 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec228b226f32a3d2-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.504 DEBUG:boto:path=/sbnvju-16/obj15 2024-03-04T16:19:15.504 DEBUG:boto:auth_path=/sbnvju-16/obj15 2024-03-04T16:19:15.504 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.504 DEBUG:boto:Path: /sbnvju-16/obj15 2024-03-04T16:19:15.505 DEBUG:boto:Data: 2024-03-04T16:19:15.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.505 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.505 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.505 DEBUG:boto:Params: {} 2024-03-04T16:19:15.505 DEBUG:boto:Token: None 2024-03-04T16:19:15.505 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj15 2024-03-04T16:19:15.505 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QPPJeUd91TIVfizHtFPMBWGT6c0= 2024-03-04T16:19:15.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QPPJeUd91TIVfizHtFPMBWGT6c0='} 2024-03-04T16:19:15.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd0ac211c66e705d-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.510 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.510 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.510 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.510 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.510 DEBUG:boto:Data: 2024-03-04T16:19:15.510 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.511 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.511 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.511 DEBUG:boto:Params: {} 2024-03-04T16:19:15.511 DEBUG:boto:Token: None 2024-03-04T16:19:15.511 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000237f7aedfec1fa81-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.513 DEBUG:boto:path=/sbnvju-16/obj16 2024-03-04T16:19:15.513 DEBUG:boto:auth_path=/sbnvju-16/obj16 2024-03-04T16:19:15.513 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.513 DEBUG:boto:Path: /sbnvju-16/obj16 2024-03-04T16:19:15.513 DEBUG:boto:Data: 2024-03-04T16:19:15.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.514 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.514 DEBUG:boto:Params: {} 2024-03-04T16:19:15.514 DEBUG:boto:Token: None 2024-03-04T16:19:15.514 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj16 2024-03-04T16:19:15.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g1ag2/RP3i75IE4pg+6BtLtF/zM= 2024-03-04T16:19:15.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g1ag2/RP3i75IE4pg+6BtLtF/zM='} 2024-03-04T16:19:15.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094c6f1574f03edf1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.561 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.561 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.561 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.561 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.561 DEBUG:boto:Data: 2024-03-04T16:19:15.561 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.561 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.561 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.561 DEBUG:boto:Params: {} 2024-03-04T16:19:15.561 DEBUG:boto:Token: None 2024-03-04T16:19:15.562 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113708b1235b6978-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.564 DEBUG:boto:path=/sbnvju-16/obj17 2024-03-04T16:19:15.564 DEBUG:boto:auth_path=/sbnvju-16/obj17 2024-03-04T16:19:15.564 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.565 DEBUG:boto:Path: /sbnvju-16/obj17 2024-03-04T16:19:15.565 DEBUG:boto:Data: 2024-03-04T16:19:15.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.565 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.565 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.565 DEBUG:boto:Params: {} 2024-03-04T16:19:15.565 DEBUG:boto:Token: None 2024-03-04T16:19:15.565 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj17 2024-03-04T16:19:15.565 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4EcFgb2fBWUfB1vPvAXtfsEgp0g= 2024-03-04T16:19:15.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4EcFgb2fBWUfB1vPvAXtfsEgp0g='} 2024-03-04T16:19:15.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f06f19025a03dc0-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.569 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.569 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.569 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.569 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.569 DEBUG:boto:Data: 2024-03-04T16:19:15.570 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.570 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.570 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.570 DEBUG:boto:Params: {} 2024-03-04T16:19:15.570 DEBUG:boto:Token: None 2024-03-04T16:19:15.570 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5e0bdc2b414bd2e-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.572 DEBUG:boto:path=/sbnvju-16/obj18 2024-03-04T16:19:15.572 DEBUG:boto:auth_path=/sbnvju-16/obj18 2024-03-04T16:19:15.572 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.572 DEBUG:boto:Path: /sbnvju-16/obj18 2024-03-04T16:19:15.572 DEBUG:boto:Data: 2024-03-04T16:19:15.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.573 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.573 DEBUG:boto:Params: {} 2024-03-04T16:19:15.573 DEBUG:boto:Token: None 2024-03-04T16:19:15.573 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj18 2024-03-04T16:19:15.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aHvzkTHhtgtEAfOgQyCdxvKv478= 2024-03-04T16:19:15.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHvzkTHhtgtEAfOgQyCdxvKv478='} 2024-03-04T16:19:15.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7bc8eba198e4003-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.620 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.621 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.621 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.621 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.621 DEBUG:boto:Data: 2024-03-04T16:19:15.621 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.621 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.621 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.621 DEBUG:boto:Params: {} 2024-03-04T16:19:15.621 DEBUG:boto:Token: None 2024-03-04T16:19:15.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.624 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c6f6b449c1b6ae4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.624 DEBUG:boto:path=/sbnvju-16/obj19 2024-03-04T16:19:15.624 DEBUG:boto:auth_path=/sbnvju-16/obj19 2024-03-04T16:19:15.624 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.624 DEBUG:boto:Path: /sbnvju-16/obj19 2024-03-04T16:19:15.624 DEBUG:boto:Data: 2024-03-04T16:19:15.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.624 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.625 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.625 DEBUG:boto:Params: {} 2024-03-04T16:19:15.625 DEBUG:boto:Token: None 2024-03-04T16:19:15.625 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj19 2024-03-04T16:19:15.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LBl3N0j4Vj2Jmv4TfWE/LqZbn9I= 2024-03-04T16:19:15.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LBl3N0j4Vj2Jmv4TfWE/LqZbn9I='} 2024-03-04T16:19:15.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010a847206d0a83a1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.629 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.629 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.629 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.629 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.629 DEBUG:boto:Data: 2024-03-04T16:19:15.629 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.629 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.629 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.629 DEBUG:boto:Params: {} 2024-03-04T16:19:15.629 DEBUG:boto:Token: None 2024-03-04T16:19:15.630 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009333bcbf868cf063-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.632 DEBUG:boto:path=/sbnvju-16/obj20 2024-03-04T16:19:15.632 DEBUG:boto:auth_path=/sbnvju-16/obj20 2024-03-04T16:19:15.632 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.632 DEBUG:boto:Path: /sbnvju-16/obj20 2024-03-04T16:19:15.632 DEBUG:boto:Data: 2024-03-04T16:19:15.632 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.632 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.632 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.632 DEBUG:boto:Params: {} 2024-03-04T16:19:15.633 DEBUG:boto:Token: None 2024-03-04T16:19:15.633 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj20 2024-03-04T16:19:15.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mZMKfjS22hQTt26ZcVjW8HEL4t0= 2024-03-04T16:19:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mZMKfjS22hQTt26ZcVjW8HEL4t0='} 2024-03-04T16:19:15.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070023ef686e011e3-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.638 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.638 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.638 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.638 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.638 DEBUG:boto:Data: 2024-03-04T16:19:15.638 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.638 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.638 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.639 DEBUG:boto:Params: {} 2024-03-04T16:19:15.639 DEBUG:boto:Token: None 2024-03-04T16:19:15.639 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.639 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf63e5da118e300-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.641 DEBUG:boto:path=/sbnvju-16/obj21 2024-03-04T16:19:15.641 DEBUG:boto:auth_path=/sbnvju-16/obj21 2024-03-04T16:19:15.641 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.641 DEBUG:boto:Path: /sbnvju-16/obj21 2024-03-04T16:19:15.641 DEBUG:boto:Data: 2024-03-04T16:19:15.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.642 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.642 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.642 DEBUG:boto:Params: {} 2024-03-04T16:19:15.642 DEBUG:boto:Token: None 2024-03-04T16:19:15.642 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj21 2024-03-04T16:19:15.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UledhRQPW5YoMLeO5XWOnx4YDrU= 2024-03-04T16:19:15.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UledhRQPW5YoMLeO5XWOnx4YDrU='} 2024-03-04T16:19:15.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e363b62fc134171-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.684 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.684 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.685 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.685 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.685 DEBUG:boto:Data: 2024-03-04T16:19:15.685 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.685 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.685 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.685 DEBUG:boto:Params: {} 2024-03-04T16:19:15.685 DEBUG:boto:Token: None 2024-03-04T16:19:15.685 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.685 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b481b12b0b28fdc5-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.688 DEBUG:boto:path=/sbnvju-16/obj22 2024-03-04T16:19:15.688 DEBUG:boto:auth_path=/sbnvju-16/obj22 2024-03-04T16:19:15.688 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.688 DEBUG:boto:Path: /sbnvju-16/obj22 2024-03-04T16:19:15.688 DEBUG:boto:Data: 2024-03-04T16:19:15.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.688 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.689 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.689 DEBUG:boto:Params: {} 2024-03-04T16:19:15.689 DEBUG:boto:Token: None 2024-03-04T16:19:15.689 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj22 2024-03-04T16:19:15.689 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBRsTZCTscqKQZc5TF+1jmNJg4k= 2024-03-04T16:19:15.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBRsTZCTscqKQZc5TF+1jmNJg4k='} 2024-03-04T16:19:15.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000607fba4dc1432e20-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.693 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.693 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.693 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.693 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.693 DEBUG:boto:Data: 2024-03-04T16:19:15.693 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.693 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.693 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.693 DEBUG:boto:Params: {} 2024-03-04T16:19:15.693 DEBUG:boto:Token: None 2024-03-04T16:19:15.693 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82be12f50ed8265-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.696 DEBUG:boto:path=/sbnvju-16/obj23 2024-03-04T16:19:15.696 DEBUG:boto:auth_path=/sbnvju-16/obj23 2024-03-04T16:19:15.696 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.696 DEBUG:boto:Path: /sbnvju-16/obj23 2024-03-04T16:19:15.696 DEBUG:boto:Data: 2024-03-04T16:19:15.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.696 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.696 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.696 DEBUG:boto:Params: {} 2024-03-04T16:19:15.697 DEBUG:boto:Token: None 2024-03-04T16:19:15.697 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj23 2024-03-04T16:19:15.697 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Yb+uqtaX7QuP1b7voMZEYPh1B+0= 2024-03-04T16:19:15.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Yb+uqtaX7QuP1b7voMZEYPh1B+0='} 2024-03-04T16:19:15.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a3c4d59b1052fa4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.700 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.700 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.701 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.701 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.701 DEBUG:boto:Data: 2024-03-04T16:19:15.701 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.701 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.701 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.701 DEBUG:boto:Params: {} 2024-03-04T16:19:15.701 DEBUG:boto:Token: None 2024-03-04T16:19:15.701 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.701 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd54920a813065ed-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.703 DEBUG:boto:path=/sbnvju-16/obj24 2024-03-04T16:19:15.704 DEBUG:boto:auth_path=/sbnvju-16/obj24 2024-03-04T16:19:15.704 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.704 DEBUG:boto:Path: /sbnvju-16/obj24 2024-03-04T16:19:15.704 DEBUG:boto:Data: 2024-03-04T16:19:15.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.704 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.704 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.704 DEBUG:boto:Params: {} 2024-03-04T16:19:15.704 DEBUG:boto:Token: None 2024-03-04T16:19:15.704 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj24 2024-03-04T16:19:15.705 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4CMGcXDgVcb8u6ZY5VhFgPeQcBQ= 2024-03-04T16:19:15.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4CMGcXDgVcb8u6ZY5VhFgPeQcBQ='} 2024-03-04T16:19:15.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009499729bfa03590f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.753 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.753 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.753 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.753 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.753 DEBUG:boto:Data: 2024-03-04T16:19:15.753 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.753 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.753 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.753 DEBUG:boto:Params: {} 2024-03-04T16:19:15.753 DEBUG:boto:Token: None 2024-03-04T16:19:15.754 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.754 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eca8e2f1d4f734d7-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.756 DEBUG:boto:path=/sbnvju-16/obj25 2024-03-04T16:19:15.756 DEBUG:boto:auth_path=/sbnvju-16/obj25 2024-03-04T16:19:15.756 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.756 DEBUG:boto:Path: /sbnvju-16/obj25 2024-03-04T16:19:15.756 DEBUG:boto:Data: 2024-03-04T16:19:15.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.757 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.757 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.757 DEBUG:boto:Params: {} 2024-03-04T16:19:15.757 DEBUG:boto:Token: None 2024-03-04T16:19:15.757 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj25 2024-03-04T16:19:15.757 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZSVtlg79kWOyqoz1+Dt4IduVGCc= 2024-03-04T16:19:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZSVtlg79kWOyqoz1+Dt4IduVGCc='} 2024-03-04T16:19:15.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f147022014a0c6df-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.761 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.761 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.761 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.761 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.761 DEBUG:boto:Data: 2024-03-04T16:19:15.761 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.762 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.762 DEBUG:boto:Params: {} 2024-03-04T16:19:15.762 DEBUG:boto:Token: None 2024-03-04T16:19:15.762 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.762 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f657efee3538427-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.764 DEBUG:boto:path=/sbnvju-16/obj26 2024-03-04T16:19:15.764 DEBUG:boto:auth_path=/sbnvju-16/obj26 2024-03-04T16:19:15.764 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.764 DEBUG:boto:Path: /sbnvju-16/obj26 2024-03-04T16:19:15.765 DEBUG:boto:Data: 2024-03-04T16:19:15.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.765 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.765 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.765 DEBUG:boto:Params: {} 2024-03-04T16:19:15.765 DEBUG:boto:Token: None 2024-03-04T16:19:15.765 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj26 2024-03-04T16:19:15.765 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tx9y2TVBm1AvPxu9qGMiHBh/A9A= 2024-03-04T16:19:15.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tx9y2TVBm1AvPxu9qGMiHBh/A9A='} 2024-03-04T16:19:15.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d747d9d8ff427109-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.808 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.808 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.809 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.809 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.809 DEBUG:boto:Data: 2024-03-04T16:19:15.809 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.809 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.809 DEBUG:boto:Params: {} 2024-03-04T16:19:15.809 DEBUG:boto:Token: None 2024-03-04T16:19:15.809 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000349634e8907efdf1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.812 DEBUG:boto:path=/sbnvju-16/obj27 2024-03-04T16:19:15.812 DEBUG:boto:auth_path=/sbnvju-16/obj27 2024-03-04T16:19:15.812 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.812 DEBUG:boto:Path: /sbnvju-16/obj27 2024-03-04T16:19:15.812 DEBUG:boto:Data: 2024-03-04T16:19:15.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.812 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.813 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.813 DEBUG:boto:Params: {} 2024-03-04T16:19:15.813 DEBUG:boto:Token: None 2024-03-04T16:19:15.813 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj27 2024-03-04T16:19:15.813 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wkj3ob31I8iHLaV4tnAyCl+UWAU= 2024-03-04T16:19:15.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wkj3ob31I8iHLaV4tnAyCl+UWAU='} 2024-03-04T16:19:15.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e64c47559a94b4a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.817 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.817 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.817 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.817 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.817 DEBUG:boto:Data: 2024-03-04T16:19:15.817 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.817 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.817 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.817 DEBUG:boto:Params: {} 2024-03-04T16:19:15.817 DEBUG:boto:Token: None 2024-03-04T16:19:15.818 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.818 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005083e714c561e441-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.820 DEBUG:boto:path=/sbnvju-16/obj28 2024-03-04T16:19:15.820 DEBUG:boto:auth_path=/sbnvju-16/obj28 2024-03-04T16:19:15.820 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.820 DEBUG:boto:Path: /sbnvju-16/obj28 2024-03-04T16:19:15.820 DEBUG:boto:Data: 2024-03-04T16:19:15.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.820 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.820 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.821 DEBUG:boto:Params: {} 2024-03-04T16:19:15.821 DEBUG:boto:Token: None 2024-03-04T16:19:15.821 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj28 2024-03-04T16:19:15.821 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DviHUXHJFK+0InMMzseg9BoJTgw= 2024-03-04T16:19:15.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DviHUXHJFK+0InMMzseg9BoJTgw='} 2024-03-04T16:19:15.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029ef902db607a3f6-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.824 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.824 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.825 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.825 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.825 DEBUG:boto:Data: 2024-03-04T16:19:15.825 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.825 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.825 DEBUG:boto:Params: {} 2024-03-04T16:19:15.825 DEBUG:boto:Token: None 2024-03-04T16:19:15.825 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.825 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007efe3df3cf338f09-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.827 DEBUG:boto:path=/sbnvju-16/obj29 2024-03-04T16:19:15.828 DEBUG:boto:auth_path=/sbnvju-16/obj29 2024-03-04T16:19:15.828 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.828 DEBUG:boto:Path: /sbnvju-16/obj29 2024-03-04T16:19:15.828 DEBUG:boto:Data: 2024-03-04T16:19:15.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.828 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.828 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.828 DEBUG:boto:Params: {} 2024-03-04T16:19:15.828 DEBUG:boto:Token: None 2024-03-04T16:19:15.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj29 2024-03-04T16:19:15.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vYZ9BYsnkVbFycgqGgSGUYKuLyc= 2024-03-04T16:19:15.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vYZ9BYsnkVbFycgqGgSGUYKuLyc='} 2024-03-04T16:19:15.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e96cab60153223a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.873 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.873 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.873 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.873 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.873 DEBUG:boto:Data: 2024-03-04T16:19:15.873 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.873 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.873 DEBUG:boto:Params: {} 2024-03-04T16:19:15.873 DEBUG:boto:Token: None 2024-03-04T16:19:15.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000168eb9600e1e0e89-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.876 DEBUG:boto:path=/sbnvju-16/obj30 2024-03-04T16:19:15.876 DEBUG:boto:auth_path=/sbnvju-16/obj30 2024-03-04T16:19:15.876 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.876 DEBUG:boto:Path: /sbnvju-16/obj30 2024-03-04T16:19:15.876 DEBUG:boto:Data: 2024-03-04T16:19:15.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.876 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.876 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.877 DEBUG:boto:Params: {} 2024-03-04T16:19:15.877 DEBUG:boto:Token: None 2024-03-04T16:19:15.877 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj30 2024-03-04T16:19:15.877 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dF4SBUQcoipfsOnsqPuDs4jWs4o= 2024-03-04T16:19:15.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dF4SBUQcoipfsOnsqPuDs4jWs4o='} 2024-03-04T16:19:15.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d92d82f3912de424-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.883 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.884 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.884 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.884 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.884 DEBUG:boto:Data: 2024-03-04T16:19:15.884 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.884 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.884 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.884 DEBUG:boto:Params: {} 2024-03-04T16:19:15.884 DEBUG:boto:Token: None 2024-03-04T16:19:15.884 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.885 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001394385581ac5ef2-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.887 DEBUG:boto:path=/sbnvju-16/obj31 2024-03-04T16:19:15.887 DEBUG:boto:auth_path=/sbnvju-16/obj31 2024-03-04T16:19:15.887 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.887 DEBUG:boto:Path: /sbnvju-16/obj31 2024-03-04T16:19:15.887 DEBUG:boto:Data: 2024-03-04T16:19:15.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.888 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.888 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.888 DEBUG:boto:Params: {} 2024-03-04T16:19:15.888 DEBUG:boto:Token: None 2024-03-04T16:19:15.888 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj31 2024-03-04T16:19:15.888 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0h9i11fwUX5yQirV/okfui1fOTk= 2024-03-04T16:19:15.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0h9i11fwUX5yQirV/okfui1fOTk='} 2024-03-04T16:19:15.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005db799fc0ae01f37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.892 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.892 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.892 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.892 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.892 DEBUG:boto:Data: 2024-03-04T16:19:15.892 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.892 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.893 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.893 DEBUG:boto:Params: {} 2024-03-04T16:19:15.893 DEBUG:boto:Token: None 2024-03-04T16:19:15.893 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.893 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c394b0219f1f6749-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.895 DEBUG:boto:path=/sbnvju-16/obj32 2024-03-04T16:19:15.895 DEBUG:boto:auth_path=/sbnvju-16/obj32 2024-03-04T16:19:15.895 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.895 DEBUG:boto:Path: /sbnvju-16/obj32 2024-03-04T16:19:15.896 DEBUG:boto:Data: 2024-03-04T16:19:15.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.896 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.896 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.896 DEBUG:boto:Params: {} 2024-03-04T16:19:15.896 DEBUG:boto:Token: None 2024-03-04T16:19:15.896 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj32 2024-03-04T16:19:15.896 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XKbEaV5QSGkggluQ1BBT3YOQGkc= 2024-03-04T16:19:15.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XKbEaV5QSGkggluQ1BBT3YOQGkc='} 2024-03-04T16:19:15.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000293e261a3976ff1d-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.900 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.900 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.900 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.900 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.900 DEBUG:boto:Data: 2024-03-04T16:19:15.900 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.901 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.901 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.901 DEBUG:boto:Params: {} 2024-03-04T16:19:15.901 DEBUG:boto:Token: None 2024-03-04T16:19:15.901 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.901 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71b15fdca192506-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.903 DEBUG:boto:path=/sbnvju-16/obj33 2024-03-04T16:19:15.903 DEBUG:boto:auth_path=/sbnvju-16/obj33 2024-03-04T16:19:15.904 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.904 DEBUG:boto:Path: /sbnvju-16/obj33 2024-03-04T16:19:15.904 DEBUG:boto:Data: 2024-03-04T16:19:15.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.904 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.904 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.904 DEBUG:boto:Params: {} 2024-03-04T16:19:15.904 DEBUG:boto:Token: None 2024-03-04T16:19:15.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj33 2024-03-04T16:19:15.904 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:b5gOPVnMNOQoYJtwJzm5cjmf5P0= 2024-03-04T16:19:15.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b5gOPVnMNOQoYJtwJzm5cjmf5P0='} 2024-03-04T16:19:15.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcbe987617208b24-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.948 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.949 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.949 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.949 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.949 DEBUG:boto:Data: 2024-03-04T16:19:15.949 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.949 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.949 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.949 DEBUG:boto:Params: {} 2024-03-04T16:19:15.949 DEBUG:boto:Token: None 2024-03-04T16:19:15.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.950 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000533bc871a6db6147-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.952 DEBUG:boto:path=/sbnvju-16/obj34 2024-03-04T16:19:15.952 DEBUG:boto:auth_path=/sbnvju-16/obj34 2024-03-04T16:19:15.952 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.952 DEBUG:boto:Path: /sbnvju-16/obj34 2024-03-04T16:19:15.952 DEBUG:boto:Data: 2024-03-04T16:19:15.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.952 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.952 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.953 DEBUG:boto:Params: {} 2024-03-04T16:19:15.953 DEBUG:boto:Token: None 2024-03-04T16:19:15.953 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj34 2024-03-04T16:19:15.953 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pyuZZRH825KsuU9N64LSV/s+N/o= 2024-03-04T16:19:15.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pyuZZRH825KsuU9N64LSV/s+N/o='} 2024-03-04T16:19:15.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e55dda2753c6c1ad-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.957 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.957 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.957 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.957 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.957 DEBUG:boto:Data: 2024-03-04T16:19:15.957 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.957 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.957 DEBUG:boto:Params: {} 2024-03-04T16:19:15.958 DEBUG:boto:Token: None 2024-03-04T16:19:15.958 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082ac3e6a69bc9a37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.960 DEBUG:boto:path=/sbnvju-16/obj35 2024-03-04T16:19:15.960 DEBUG:boto:auth_path=/sbnvju-16/obj35 2024-03-04T16:19:15.960 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.960 DEBUG:boto:Path: /sbnvju-16/obj35 2024-03-04T16:19:15.960 DEBUG:boto:Data: 2024-03-04T16:19:15.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.961 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.961 DEBUG:boto:Params: {} 2024-03-04T16:19:15.961 DEBUG:boto:Token: None 2024-03-04T16:19:15.961 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj35 2024-03-04T16:19:15.961 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mAG3tK4d677+vOWczsFQDZ64e0M= 2024-03-04T16:19:15.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mAG3tK4d677+vOWczsFQDZ64e0M='} 2024-03-04T16:19:15.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8f8fe914240836c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.965 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:15.965 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:15.965 DEBUG:boto:Method: HEAD 2024-03-04T16:19:15.965 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:15.965 DEBUG:boto:Data: 2024-03-04T16:19:15.965 DEBUG:boto:Headers: {} 2024-03-04T16:19:15.965 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.966 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.966 DEBUG:boto:Params: {} 2024-03-04T16:19:15.966 DEBUG:boto:Token: None 2024-03-04T16:19:15.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/ 2024-03-04T16:19:15.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T16:19:15.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T16:19:15.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fb6ad8f25a002c1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:15.968 DEBUG:boto:path=/sbnvju-16/obj36 2024-03-04T16:19:15.968 DEBUG:boto:auth_path=/sbnvju-16/obj36 2024-03-04T16:19:15.968 DEBUG:boto:Method: PUT 2024-03-04T16:19:15.968 DEBUG:boto:Path: /sbnvju-16/obj36 2024-03-04T16:19:15.969 DEBUG:boto:Data: 2024-03-04T16:19:15.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:15.969 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:15.969 DEBUG:boto:Port: 8000 2024-03-04T16:19:15.969 DEBUG:boto:Params: {} 2024-03-04T16:19:15.969 DEBUG:boto:Token: None 2024-03-04T16:19:15.969 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:15 GMT /sbnvju-16/obj36 2024-03-04T16:19:15.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brlBzM5dw/8ikHUiv57mRbDchv4= 2024-03-04T16:19:15.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brlBzM5dw/8ikHUiv57mRbDchv4='} 2024-03-04T16:19:16.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009027de3d1241d40a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.013 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.013 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.013 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.013 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.013 DEBUG:boto:Data: 2024-03-04T16:19:16.013 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.013 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.013 DEBUG:boto:Params: {} 2024-03-04T16:19:16.013 DEBUG:boto:Token: None 2024-03-04T16:19:16.014 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000348848636c709071-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.016 DEBUG:boto:path=/sbnvju-16/obj37 2024-03-04T16:19:16.016 DEBUG:boto:auth_path=/sbnvju-16/obj37 2024-03-04T16:19:16.016 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.016 DEBUG:boto:Path: /sbnvju-16/obj37 2024-03-04T16:19:16.016 DEBUG:boto:Data: 2024-03-04T16:19:16.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.017 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.017 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.017 DEBUG:boto:Params: {} 2024-03-04T16:19:16.017 DEBUG:boto:Token: None 2024-03-04T16:19:16.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj37 2024-03-04T16:19:16.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:590epzfjK/rQUK5fYEpDyX70Uas= 2024-03-04T16:19:16.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:590epzfjK/rQUK5fYEpDyX70Uas='} 2024-03-04T16:19:16.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092afaae54b3d38d8-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.021 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.021 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.021 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.021 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.021 DEBUG:boto:Data: 2024-03-04T16:19:16.021 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.022 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.022 DEBUG:boto:Params: {} 2024-03-04T16:19:16.022 DEBUG:boto:Token: None 2024-03-04T16:19:16.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbda926a261da02a-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.024 DEBUG:boto:path=/sbnvju-16/obj38 2024-03-04T16:19:16.024 DEBUG:boto:auth_path=/sbnvju-16/obj38 2024-03-04T16:19:16.024 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.024 DEBUG:boto:Path: /sbnvju-16/obj38 2024-03-04T16:19:16.024 DEBUG:boto:Data: 2024-03-04T16:19:16.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.025 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.025 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.025 DEBUG:boto:Params: {} 2024-03-04T16:19:16.025 DEBUG:boto:Token: None 2024-03-04T16:19:16.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj38 2024-03-04T16:19:16.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uwnuZPMneQNDNIMcZVbZL6LwoU8= 2024-03-04T16:19:16.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uwnuZPMneQNDNIMcZVbZL6LwoU8='} 2024-03-04T16:19:16.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cd5e5ea0a7cfbee-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.072 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.073 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.073 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.073 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.073 DEBUG:boto:Data: 2024-03-04T16:19:16.073 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.073 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.073 DEBUG:boto:Params: {} 2024-03-04T16:19:16.073 DEBUG:boto:Token: None 2024-03-04T16:19:16.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c1a273c21c0f7c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.076 DEBUG:boto:path=/sbnvju-16/obj39 2024-03-04T16:19:16.076 DEBUG:boto:auth_path=/sbnvju-16/obj39 2024-03-04T16:19:16.076 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.076 DEBUG:boto:Path: /sbnvju-16/obj39 2024-03-04T16:19:16.076 DEBUG:boto:Data: 2024-03-04T16:19:16.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.077 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.077 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.077 DEBUG:boto:Params: {} 2024-03-04T16:19:16.077 DEBUG:boto:Token: None 2024-03-04T16:19:16.077 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj39 2024-03-04T16:19:16.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4waapvH3BJF2a6OAsFh3TxshjNk= 2024-03-04T16:19:16.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4waapvH3BJF2a6OAsFh3TxshjNk='} 2024-03-04T16:19:16.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dfd627d54a71e60-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.081 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.081 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.081 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.081 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.081 DEBUG:boto:Data: 2024-03-04T16:19:16.082 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.082 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.082 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.082 DEBUG:boto:Params: {} 2024-03-04T16:19:16.082 DEBUG:boto:Token: None 2024-03-04T16:19:16.082 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008055af68f1897ce2-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.084 DEBUG:boto:path=/sbnvju-16/obj40 2024-03-04T16:19:16.084 DEBUG:boto:auth_path=/sbnvju-16/obj40 2024-03-04T16:19:16.084 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.084 DEBUG:boto:Path: /sbnvju-16/obj40 2024-03-04T16:19:16.085 DEBUG:boto:Data: 2024-03-04T16:19:16.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.085 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.085 DEBUG:boto:Params: {} 2024-03-04T16:19:16.085 DEBUG:boto:Token: None 2024-03-04T16:19:16.085 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj40 2024-03-04T16:19:16.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TTPfBUt7hnbs4ViTPboZK4F0Q3M= 2024-03-04T16:19:16.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TTPfBUt7hnbs4ViTPboZK4F0Q3M='} 2024-03-04T16:19:16.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000992a64b8f7ce3120-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.129 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.129 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.129 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.129 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.129 DEBUG:boto:Data: 2024-03-04T16:19:16.129 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.129 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.129 DEBUG:boto:Params: {} 2024-03-04T16:19:16.129 DEBUG:boto:Token: None 2024-03-04T16:19:16.130 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013d22651c45b5faf-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.132 DEBUG:boto:path=/sbnvju-16/obj41 2024-03-04T16:19:16.132 DEBUG:boto:auth_path=/sbnvju-16/obj41 2024-03-04T16:19:16.132 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.132 DEBUG:boto:Path: /sbnvju-16/obj41 2024-03-04T16:19:16.132 DEBUG:boto:Data: 2024-03-04T16:19:16.132 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.132 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.133 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.133 DEBUG:boto:Params: {} 2024-03-04T16:19:16.133 DEBUG:boto:Token: None 2024-03-04T16:19:16.133 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj41 2024-03-04T16:19:16.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uu41hN+Gv4Xammmp8U+7VU+Fs3k= 2024-03-04T16:19:16.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uu41hN+Gv4Xammmp8U+7VU+Fs3k='} 2024-03-04T16:19:16.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6843e996e4673de-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.136 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.137 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.137 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.137 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.137 DEBUG:boto:Data: 2024-03-04T16:19:16.137 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.137 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.137 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.137 DEBUG:boto:Params: {} 2024-03-04T16:19:16.137 DEBUG:boto:Token: None 2024-03-04T16:19:16.137 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f82cb6569e40d66-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.140 DEBUG:boto:path=/sbnvju-16/obj42 2024-03-04T16:19:16.140 DEBUG:boto:auth_path=/sbnvju-16/obj42 2024-03-04T16:19:16.140 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.140 DEBUG:boto:Path: /sbnvju-16/obj42 2024-03-04T16:19:16.140 DEBUG:boto:Data: 2024-03-04T16:19:16.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.140 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.140 DEBUG:boto:Params: {} 2024-03-04T16:19:16.140 DEBUG:boto:Token: None 2024-03-04T16:19:16.141 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj42 2024-03-04T16:19:16.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QiPQf6sYZQh4GMN3/zDhSNdC+0M= 2024-03-04T16:19:16.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QiPQf6sYZQh4GMN3/zDhSNdC+0M='} 2024-03-04T16:19:16.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a84193703169a826-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.144 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.144 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.144 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.144 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.145 DEBUG:boto:Data: 2024-03-04T16:19:16.145 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.145 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.145 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.145 DEBUG:boto:Params: {} 2024-03-04T16:19:16.145 DEBUG:boto:Token: None 2024-03-04T16:19:16.145 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038d8f9b3f68007a3-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.147 DEBUG:boto:path=/sbnvju-16/obj43 2024-03-04T16:19:16.147 DEBUG:boto:auth_path=/sbnvju-16/obj43 2024-03-04T16:19:16.147 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.148 DEBUG:boto:Path: /sbnvju-16/obj43 2024-03-04T16:19:16.148 DEBUG:boto:Data: 2024-03-04T16:19:16.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.148 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.148 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.148 DEBUG:boto:Params: {} 2024-03-04T16:19:16.148 DEBUG:boto:Token: None 2024-03-04T16:19:16.148 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj43 2024-03-04T16:19:16.148 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2a4zWh3P0b4He2qqljk8VMve9Us= 2024-03-04T16:19:16.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2a4zWh3P0b4He2qqljk8VMve9Us='} 2024-03-04T16:19:16.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000406617fedbcb73b7-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.196 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.197 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.197 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.197 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.197 DEBUG:boto:Data: 2024-03-04T16:19:16.197 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.197 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.197 DEBUG:boto:Params: {} 2024-03-04T16:19:16.197 DEBUG:boto:Token: None 2024-03-04T16:19:16.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3cf1addb4a0c7ec-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.200 DEBUG:boto:path=/sbnvju-16/obj44 2024-03-04T16:19:16.200 DEBUG:boto:auth_path=/sbnvju-16/obj44 2024-03-04T16:19:16.200 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.201 DEBUG:boto:Path: /sbnvju-16/obj44 2024-03-04T16:19:16.201 DEBUG:boto:Data: 2024-03-04T16:19:16.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.201 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.201 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.201 DEBUG:boto:Params: {} 2024-03-04T16:19:16.201 DEBUG:boto:Token: None 2024-03-04T16:19:16.201 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj44 2024-03-04T16:19:16.201 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zzBMIdwsx6oM5/1PTMvTIFeJQ88= 2024-03-04T16:19:16.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zzBMIdwsx6oM5/1PTMvTIFeJQ88='} 2024-03-04T16:19:16.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067205b98de3b359a-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.206 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.206 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.206 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.206 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.207 DEBUG:boto:Data: 2024-03-04T16:19:16.207 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.207 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.207 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.207 DEBUG:boto:Params: {} 2024-03-04T16:19:16.207 DEBUG:boto:Token: None 2024-03-04T16:19:16.207 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.207 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000658f6a993f02cba4-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.210 DEBUG:boto:path=/sbnvju-16/obj45 2024-03-04T16:19:16.210 DEBUG:boto:auth_path=/sbnvju-16/obj45 2024-03-04T16:19:16.210 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.210 DEBUG:boto:Path: /sbnvju-16/obj45 2024-03-04T16:19:16.210 DEBUG:boto:Data: 2024-03-04T16:19:16.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.210 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.210 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.210 DEBUG:boto:Params: {} 2024-03-04T16:19:16.210 DEBUG:boto:Token: None 2024-03-04T16:19:16.211 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj45 2024-03-04T16:19:16.211 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t1F7wks54+26W5jEI2zAmqA6yww= 2024-03-04T16:19:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t1F7wks54+26W5jEI2zAmqA6yww='} 2024-03-04T16:19:16.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000825f2f9664dfd4a0-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.252 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.253 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.253 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.253 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.253 DEBUG:boto:Data: 2024-03-04T16:19:16.253 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.253 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.253 DEBUG:boto:Params: {} 2024-03-04T16:19:16.253 DEBUG:boto:Token: None 2024-03-04T16:19:16.253 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b47377a53fb1d22-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.256 DEBUG:boto:path=/sbnvju-16/obj46 2024-03-04T16:19:16.256 DEBUG:boto:auth_path=/sbnvju-16/obj46 2024-03-04T16:19:16.256 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.256 DEBUG:boto:Path: /sbnvju-16/obj46 2024-03-04T16:19:16.256 DEBUG:boto:Data: 2024-03-04T16:19:16.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.256 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.256 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.257 DEBUG:boto:Params: {} 2024-03-04T16:19:16.257 DEBUG:boto:Token: None 2024-03-04T16:19:16.257 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj46 2024-03-04T16:19:16.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ev+UtUIfghsnxv7Y/Kr/araCJZ0= 2024-03-04T16:19:16.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ev+UtUIfghsnxv7Y/Kr/araCJZ0='} 2024-03-04T16:19:16.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acfde32f6ca7b8dd-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.260 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.261 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.261 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.261 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.261 DEBUG:boto:Data: 2024-03-04T16:19:16.261 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.261 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.261 DEBUG:boto:Params: {} 2024-03-04T16:19:16.261 DEBUG:boto:Token: None 2024-03-04T16:19:16.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.262 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0697f821ac1b76e-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.264 DEBUG:boto:path=/sbnvju-16/obj47 2024-03-04T16:19:16.264 DEBUG:boto:auth_path=/sbnvju-16/obj47 2024-03-04T16:19:16.264 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.264 DEBUG:boto:Path: /sbnvju-16/obj47 2024-03-04T16:19:16.264 DEBUG:boto:Data: 2024-03-04T16:19:16.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.264 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.264 DEBUG:boto:Params: {} 2024-03-04T16:19:16.264 DEBUG:boto:Token: None 2024-03-04T16:19:16.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj47 2024-03-04T16:19:16.265 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zGg0ymRHkYFYaA+SQ8mbzLmoGpE= 2024-03-04T16:19:16.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zGg0ymRHkYFYaA+SQ8mbzLmoGpE='} 2024-03-04T16:19:16.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000519569b88a4b531c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.268 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.268 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.268 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.268 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.268 DEBUG:boto:Data: 2024-03-04T16:19:16.268 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.269 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.269 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.269 DEBUG:boto:Params: {} 2024-03-04T16:19:16.269 DEBUG:boto:Token: None 2024-03-04T16:19:16.269 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.269 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5ae95673aeea280-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.271 DEBUG:boto:path=/sbnvju-16/obj48 2024-03-04T16:19:16.271 DEBUG:boto:auth_path=/sbnvju-16/obj48 2024-03-04T16:19:16.271 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.271 DEBUG:boto:Path: /sbnvju-16/obj48 2024-03-04T16:19:16.271 DEBUG:boto:Data: 2024-03-04T16:19:16.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.272 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.272 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.272 DEBUG:boto:Params: {} 2024-03-04T16:19:16.272 DEBUG:boto:Token: None 2024-03-04T16:19:16.272 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj48 2024-03-04T16:19:16.272 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5DB52MqYofpLT+pV9YsoqJCYiSo= 2024-03-04T16:19:16.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5DB52MqYofpLT+pV9YsoqJCYiSo='} 2024-03-04T16:19:16.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9787bb56425021c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.321 DEBUG:boto:path=/sbnvju-16/ 2024-03-04T16:19:16.321 DEBUG:boto:auth_path=/sbnvju-16/ 2024-03-04T16:19:16.321 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.321 DEBUG:boto:Path: /sbnvju-16/ 2024-03-04T16:19:16.321 DEBUG:boto:Data: 2024-03-04T16:19:16.321 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.321 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.321 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.321 DEBUG:boto:Params: {} 2024-03-04T16:19:16.321 DEBUG:boto:Token: None 2024-03-04T16:19:16.322 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/ 2024-03-04T16:19:16.322 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T16:19:16.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T16:19:16.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d1c0b0523c01422-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.324 DEBUG:boto:path=/sbnvju-16/obj49 2024-03-04T16:19:16.324 DEBUG:boto:auth_path=/sbnvju-16/obj49 2024-03-04T16:19:16.324 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.324 DEBUG:boto:Path: /sbnvju-16/obj49 2024-03-04T16:19:16.324 DEBUG:boto:Data: 2024-03-04T16:19:16.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:19:16.325 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.325 DEBUG:boto:Params: {} 2024-03-04T16:19:16.325 DEBUG:boto:Token: None 2024-03-04T16:19:16.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-16/obj49 2024-03-04T16:19:16.325 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DGhZW4+okHkQqifvLDL64Tvb+7w= 2024-03-04T16:19:16.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DGhZW4+okHkQqifvLDL64Tvb+7w='} 2024-03-04T16:19:16.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097f002373c8571ef-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.329 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.329 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.329 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.329 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.329 DEBUG:boto:Data: 2024-03-04T16:19:16.329 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.329 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.329 DEBUG:boto:Params: {} 2024-03-04T16:19:16.329 DEBUG:boto:Token: None 2024-03-04T16:19:16.330 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a9a88ff24c6633c-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.332 DEBUG:boto:path=/sbnvju-17/obj1 2024-03-04T16:19:16.333 DEBUG:boto:auth_path=/sbnvju-17/obj1 2024-03-04T16:19:16.333 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.333 DEBUG:boto:Path: /sbnvju-17/obj1 2024-03-04T16:19:16.333 DEBUG:boto:Data: 2024-03-04T16:19:16.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.333 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.333 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.333 DEBUG:boto:Params: {} 2024-03-04T16:19:16.333 DEBUG:boto:Token: None 2024-03-04T16:19:16.333 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj1 2024-03-04T16:19:16.334 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BhACC10gHBqlE8hJF2WK+rFaW04= 2024-03-04T16:19:16.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BhACC10gHBqlE8hJF2WK+rFaW04='} 2024-03-04T16:19:16.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000909607b39151743c-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.376 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.377 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.377 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.377 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.377 DEBUG:boto:Data: 2024-03-04T16:19:16.377 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.377 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.377 DEBUG:boto:Params: {} 2024-03-04T16:19:16.377 DEBUG:boto:Token: None 2024-03-04T16:19:16.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.377 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000600180f2056cc6ac-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.380 DEBUG:boto:path=/sbnvju-17/obj2 2024-03-04T16:19:16.380 DEBUG:boto:auth_path=/sbnvju-17/obj2 2024-03-04T16:19:16.380 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.380 DEBUG:boto:Path: /sbnvju-17/obj2 2024-03-04T16:19:16.381 DEBUG:boto:Data: 2024-03-04T16:19:16.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.381 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.381 DEBUG:boto:Params: {} 2024-03-04T16:19:16.381 DEBUG:boto:Token: None 2024-03-04T16:19:16.381 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj2 2024-03-04T16:19:16.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4RsDKP+xZJoVr3kWloVSK0O4+3c= 2024-03-04T16:19:16.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RsDKP+xZJoVr3kWloVSK0O4+3c='} 2024-03-04T16:19:16.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042595f32e98da6bf-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.389 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.389 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.390 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.390 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.390 DEBUG:boto:Data: 2024-03-04T16:19:16.390 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.390 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.390 DEBUG:boto:Params: {} 2024-03-04T16:19:16.390 DEBUG:boto:Token: None 2024-03-04T16:19:16.390 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003efc859bed5e36ff-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.393 DEBUG:boto:path=/sbnvju-17/obj3 2024-03-04T16:19:16.393 DEBUG:boto:auth_path=/sbnvju-17/obj3 2024-03-04T16:19:16.393 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.393 DEBUG:boto:Path: /sbnvju-17/obj3 2024-03-04T16:19:16.393 DEBUG:boto:Data: 2024-03-04T16:19:16.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.393 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.394 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.394 DEBUG:boto:Params: {} 2024-03-04T16:19:16.394 DEBUG:boto:Token: None 2024-03-04T16:19:16.394 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj3 2024-03-04T16:19:16.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lIF01ANDufqivajee0rp5TobYRQ= 2024-03-04T16:19:16.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lIF01ANDufqivajee0rp5TobYRQ='} 2024-03-04T16:19:16.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000970f76d11447b1e7-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.436 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.436 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.437 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.437 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.437 DEBUG:boto:Data: 2024-03-04T16:19:16.437 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.437 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.437 DEBUG:boto:Params: {} 2024-03-04T16:19:16.437 DEBUG:boto:Token: None 2024-03-04T16:19:16.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008acf02b00e3b7fa-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.440 DEBUG:boto:path=/sbnvju-17/obj4 2024-03-04T16:19:16.440 DEBUG:boto:auth_path=/sbnvju-17/obj4 2024-03-04T16:19:16.440 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.440 DEBUG:boto:Path: /sbnvju-17/obj4 2024-03-04T16:19:16.440 DEBUG:boto:Data: 2024-03-04T16:19:16.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.441 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.441 DEBUG:boto:Params: {} 2024-03-04T16:19:16.441 DEBUG:boto:Token: None 2024-03-04T16:19:16.441 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj4 2024-03-04T16:19:16.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ner0x96wCW2ysbwV7YqzXvabhjc= 2024-03-04T16:19:16.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ner0x96wCW2ysbwV7YqzXvabhjc='} 2024-03-04T16:19:16.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf75ce4fc0a3b6c4-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.447 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.447 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.447 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.447 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.448 DEBUG:boto:Data: 2024-03-04T16:19:16.448 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.448 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.448 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.448 DEBUG:boto:Params: {} 2024-03-04T16:19:16.448 DEBUG:boto:Token: None 2024-03-04T16:19:16.448 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.448 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bea8d9dd8458a6d-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.451 DEBUG:boto:path=/sbnvju-17/obj5 2024-03-04T16:19:16.451 DEBUG:boto:auth_path=/sbnvju-17/obj5 2024-03-04T16:19:16.451 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.451 DEBUG:boto:Path: /sbnvju-17/obj5 2024-03-04T16:19:16.451 DEBUG:boto:Data: 2024-03-04T16:19:16.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.451 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.451 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.451 DEBUG:boto:Params: {} 2024-03-04T16:19:16.451 DEBUG:boto:Token: None 2024-03-04T16:19:16.452 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj5 2024-03-04T16:19:16.452 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qFHdyqi6+wkL480pT02dS0kVVIY= 2024-03-04T16:19:16.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qFHdyqi6+wkL480pT02dS0kVVIY='} 2024-03-04T16:19:16.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bae1265fbbbb04a1-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.456 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.456 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.456 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.456 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.456 DEBUG:boto:Data: 2024-03-04T16:19:16.456 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.457 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.457 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.457 DEBUG:boto:Params: {} 2024-03-04T16:19:16.457 DEBUG:boto:Token: None 2024-03-04T16:19:16.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ce415ef912a93d9-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.460 DEBUG:boto:path=/sbnvju-17/obj6 2024-03-04T16:19:16.460 DEBUG:boto:auth_path=/sbnvju-17/obj6 2024-03-04T16:19:16.460 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.460 DEBUG:boto:Path: /sbnvju-17/obj6 2024-03-04T16:19:16.460 DEBUG:boto:Data: 2024-03-04T16:19:16.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.460 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.460 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.460 DEBUG:boto:Params: {} 2024-03-04T16:19:16.460 DEBUG:boto:Token: None 2024-03-04T16:19:16.461 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj6 2024-03-04T16:19:16.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A4sn1jrI1xzt6D628Lo85pi+nCI= 2024-03-04T16:19:16.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A4sn1jrI1xzt6D628Lo85pi+nCI='} 2024-03-04T16:19:16.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ccbca55db734084-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.465 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.465 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.465 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.465 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.465 DEBUG:boto:Data: 2024-03-04T16:19:16.465 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.466 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.466 DEBUG:boto:Params: {} 2024-03-04T16:19:16.466 DEBUG:boto:Token: None 2024-03-04T16:19:16.466 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098370d552bbf02e4-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.468 DEBUG:boto:path=/sbnvju-17/obj7 2024-03-04T16:19:16.469 DEBUG:boto:auth_path=/sbnvju-17/obj7 2024-03-04T16:19:16.469 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.469 DEBUG:boto:Path: /sbnvju-17/obj7 2024-03-04T16:19:16.469 DEBUG:boto:Data: 2024-03-04T16:19:16.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.469 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.469 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.469 DEBUG:boto:Params: {} 2024-03-04T16:19:16.469 DEBUG:boto:Token: None 2024-03-04T16:19:16.469 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj7 2024-03-04T16:19:16.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rA9Psq8fLyEVearUvsrbsr7OUXM= 2024-03-04T16:19:16.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rA9Psq8fLyEVearUvsrbsr7OUXM='} 2024-03-04T16:19:16.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ebc1701022580a5-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.516 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.516 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.516 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.517 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.517 DEBUG:boto:Data: 2024-03-04T16:19:16.517 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.517 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.517 DEBUG:boto:Params: {} 2024-03-04T16:19:16.517 DEBUG:boto:Token: None 2024-03-04T16:19:16.517 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008267478a8b78ded6-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.520 DEBUG:boto:path=/sbnvju-17/obj8 2024-03-04T16:19:16.520 DEBUG:boto:auth_path=/sbnvju-17/obj8 2024-03-04T16:19:16.520 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.520 DEBUG:boto:Path: /sbnvju-17/obj8 2024-03-04T16:19:16.520 DEBUG:boto:Data: 2024-03-04T16:19:16.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.520 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.520 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.521 DEBUG:boto:Params: {} 2024-03-04T16:19:16.521 DEBUG:boto:Token: None 2024-03-04T16:19:16.521 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj8 2024-03-04T16:19:16.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECaQfC1bBtQ3EskNck2TFihM+SQ= 2024-03-04T16:19:16.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECaQfC1bBtQ3EskNck2TFihM+SQ='} 2024-03-04T16:19:16.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000943c644dc0cea19-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.526 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.526 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.526 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.526 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.526 DEBUG:boto:Data: 2024-03-04T16:19:16.526 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.526 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.526 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.526 DEBUG:boto:Params: {} 2024-03-04T16:19:16.526 DEBUG:boto:Token: None 2024-03-04T16:19:16.526 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c6dbf3ce096e409-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.529 DEBUG:boto:path=/sbnvju-17/obj9 2024-03-04T16:19:16.529 DEBUG:boto:auth_path=/sbnvju-17/obj9 2024-03-04T16:19:16.529 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.529 DEBUG:boto:Path: /sbnvju-17/obj9 2024-03-04T16:19:16.529 DEBUG:boto:Data: 2024-03-04T16:19:16.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.529 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.530 DEBUG:boto:Params: {} 2024-03-04T16:19:16.530 DEBUG:boto:Token: None 2024-03-04T16:19:16.530 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj9 2024-03-04T16:19:16.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jNKhV8edSyFZwSH4KOSKh+ACpCo= 2024-03-04T16:19:16.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jNKhV8edSyFZwSH4KOSKh+ACpCo='} 2024-03-04T16:19:16.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0191189fb3c15d5-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.572 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.572 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.573 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.573 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.573 DEBUG:boto:Data: 2024-03-04T16:19:16.573 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.573 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.573 DEBUG:boto:Params: {} 2024-03-04T16:19:16.573 DEBUG:boto:Token: None 2024-03-04T16:19:16.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb7b7c316242084e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.576 DEBUG:boto:path=/sbnvju-17/obj10 2024-03-04T16:19:16.576 DEBUG:boto:auth_path=/sbnvju-17/obj10 2024-03-04T16:19:16.576 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.576 DEBUG:boto:Path: /sbnvju-17/obj10 2024-03-04T16:19:16.576 DEBUG:boto:Data: 2024-03-04T16:19:16.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.576 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.576 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.576 DEBUG:boto:Params: {} 2024-03-04T16:19:16.577 DEBUG:boto:Token: None 2024-03-04T16:19:16.577 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj10 2024-03-04T16:19:16.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1+RWjymuPOmKubEkk2HSRI01ai8= 2024-03-04T16:19:16.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1+RWjymuPOmKubEkk2HSRI01ai8='} 2024-03-04T16:19:16.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9c742b50abea2a8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.582 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.582 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.582 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.582 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.582 DEBUG:boto:Data: 2024-03-04T16:19:16.582 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.582 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.582 DEBUG:boto:Params: {} 2024-03-04T16:19:16.583 DEBUG:boto:Token: None 2024-03-04T16:19:16.583 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.583 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dbd72cbca835c30-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.585 DEBUG:boto:path=/sbnvju-17/obj11 2024-03-04T16:19:16.585 DEBUG:boto:auth_path=/sbnvju-17/obj11 2024-03-04T16:19:16.585 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.585 DEBUG:boto:Path: /sbnvju-17/obj11 2024-03-04T16:19:16.586 DEBUG:boto:Data: 2024-03-04T16:19:16.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.586 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.586 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.586 DEBUG:boto:Params: {} 2024-03-04T16:19:16.586 DEBUG:boto:Token: None 2024-03-04T16:19:16.586 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj11 2024-03-04T16:19:16.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9DkqnSUiz2s5DcyVwkKm1XKLXeQ= 2024-03-04T16:19:16.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9DkqnSUiz2s5DcyVwkKm1XKLXeQ='} 2024-03-04T16:19:16.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfb622ea1e86e91f-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.628 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.628 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.629 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.629 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.629 DEBUG:boto:Data: 2024-03-04T16:19:16.629 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.629 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.629 DEBUG:boto:Params: {} 2024-03-04T16:19:16.629 DEBUG:boto:Token: None 2024-03-04T16:19:16.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.629 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042ae9b6c20027c36-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.632 DEBUG:boto:path=/sbnvju-17/obj12 2024-03-04T16:19:16.632 DEBUG:boto:auth_path=/sbnvju-17/obj12 2024-03-04T16:19:16.632 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.632 DEBUG:boto:Path: /sbnvju-17/obj12 2024-03-04T16:19:16.632 DEBUG:boto:Data: 2024-03-04T16:19:16.632 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.632 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.633 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.633 DEBUG:boto:Params: {} 2024-03-04T16:19:16.633 DEBUG:boto:Token: None 2024-03-04T16:19:16.633 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj12 2024-03-04T16:19:16.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kq1dDRe+W0oz4FP5O6SUhKe7cZg= 2024-03-04T16:19:16.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kq1dDRe+W0oz4FP5O6SUhKe7cZg='} 2024-03-04T16:19:16.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cf96ce862d6d0da-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.637 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.637 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.637 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.637 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.637 DEBUG:boto:Data: 2024-03-04T16:19:16.638 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.638 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.638 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.638 DEBUG:boto:Params: {} 2024-03-04T16:19:16.638 DEBUG:boto:Token: None 2024-03-04T16:19:16.638 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b359c176c60db81-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.640 DEBUG:boto:path=/sbnvju-17/obj13 2024-03-04T16:19:16.641 DEBUG:boto:auth_path=/sbnvju-17/obj13 2024-03-04T16:19:16.641 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.641 DEBUG:boto:Path: /sbnvju-17/obj13 2024-03-04T16:19:16.641 DEBUG:boto:Data: 2024-03-04T16:19:16.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.641 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.641 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.641 DEBUG:boto:Params: {} 2024-03-04T16:19:16.641 DEBUG:boto:Token: None 2024-03-04T16:19:16.641 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj13 2024-03-04T16:19:16.641 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5hdS4dSdcZXoFdH5pOEHg7kuh4E= 2024-03-04T16:19:16.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5hdS4dSdcZXoFdH5pOEHg7kuh4E='} 2024-03-04T16:19:16.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066d46ecb6e1bdb0b-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.684 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.684 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.684 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.685 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.685 DEBUG:boto:Data: 2024-03-04T16:19:16.685 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.685 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.685 DEBUG:boto:Params: {} 2024-03-04T16:19:16.685 DEBUG:boto:Token: None 2024-03-04T16:19:16.685 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.685 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220afa5b75499584-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.688 DEBUG:boto:path=/sbnvju-17/obj14 2024-03-04T16:19:16.688 DEBUG:boto:auth_path=/sbnvju-17/obj14 2024-03-04T16:19:16.688 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.688 DEBUG:boto:Path: /sbnvju-17/obj14 2024-03-04T16:19:16.688 DEBUG:boto:Data: 2024-03-04T16:19:16.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.688 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.688 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.688 DEBUG:boto:Params: {} 2024-03-04T16:19:16.688 DEBUG:boto:Token: None 2024-03-04T16:19:16.689 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj14 2024-03-04T16:19:16.689 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LF3FcclFGl+OnBZT0FKkeuR/CKs= 2024-03-04T16:19:16.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LF3FcclFGl+OnBZT0FKkeuR/CKs='} 2024-03-04T16:19:16.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cdc09e610ac3fcb-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.693 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.693 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.693 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.693 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.693 DEBUG:boto:Data: 2024-03-04T16:19:16.693 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.694 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.694 DEBUG:boto:Params: {} 2024-03-04T16:19:16.694 DEBUG:boto:Token: None 2024-03-04T16:19:16.694 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029b465d5de429fef-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.696 DEBUG:boto:path=/sbnvju-17/obj15 2024-03-04T16:19:16.697 DEBUG:boto:auth_path=/sbnvju-17/obj15 2024-03-04T16:19:16.697 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.697 DEBUG:boto:Path: /sbnvju-17/obj15 2024-03-04T16:19:16.697 DEBUG:boto:Data: 2024-03-04T16:19:16.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.697 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.697 DEBUG:boto:Params: {} 2024-03-04T16:19:16.697 DEBUG:boto:Token: None 2024-03-04T16:19:16.697 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj15 2024-03-04T16:19:16.697 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a6OoBFADN/s7xB9o3ktkPoMvzIw= 2024-03-04T16:19:16.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a6OoBFADN/s7xB9o3ktkPoMvzIw='} 2024-03-04T16:19:16.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c4de98c0d088526-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.741 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.741 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.741 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.741 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.741 DEBUG:boto:Data: 2024-03-04T16:19:16.741 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.741 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.741 DEBUG:boto:Params: {} 2024-03-04T16:19:16.742 DEBUG:boto:Token: None 2024-03-04T16:19:16.742 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.742 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9d9053406688cf9-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.744 DEBUG:boto:path=/sbnvju-17/obj16 2024-03-04T16:19:16.745 DEBUG:boto:auth_path=/sbnvju-17/obj16 2024-03-04T16:19:16.745 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.745 DEBUG:boto:Path: /sbnvju-17/obj16 2024-03-04T16:19:16.745 DEBUG:boto:Data: 2024-03-04T16:19:16.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.745 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.745 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.745 DEBUG:boto:Params: {} 2024-03-04T16:19:16.745 DEBUG:boto:Token: None 2024-03-04T16:19:16.745 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj16 2024-03-04T16:19:16.745 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:evcclatReXiLduZaXdb7ytVvACk= 2024-03-04T16:19:16.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:evcclatReXiLduZaXdb7ytVvACk='} 2024-03-04T16:19:16.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e50b37df72d9422e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.754 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.754 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.754 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.754 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.754 DEBUG:boto:Data: 2024-03-04T16:19:16.754 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.754 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.754 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.755 DEBUG:boto:Params: {} 2024-03-04T16:19:16.755 DEBUG:boto:Token: None 2024-03-04T16:19:16.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006eec93429fbf52be-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.757 DEBUG:boto:path=/sbnvju-17/obj17 2024-03-04T16:19:16.757 DEBUG:boto:auth_path=/sbnvju-17/obj17 2024-03-04T16:19:16.757 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.758 DEBUG:boto:Path: /sbnvju-17/obj17 2024-03-04T16:19:16.758 DEBUG:boto:Data: 2024-03-04T16:19:16.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.758 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.758 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.758 DEBUG:boto:Params: {} 2024-03-04T16:19:16.758 DEBUG:boto:Token: None 2024-03-04T16:19:16.758 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj17 2024-03-04T16:19:16.758 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iDM3AKbwvL000AjcN12/R47YSmc= 2024-03-04T16:19:16.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDM3AKbwvL000AjcN12/R47YSmc='} 2024-03-04T16:19:16.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fbc5e4250b111fd-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.800 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.801 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.801 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.801 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.801 DEBUG:boto:Data: 2024-03-04T16:19:16.801 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.801 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.801 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.801 DEBUG:boto:Params: {} 2024-03-04T16:19:16.801 DEBUG:boto:Token: None 2024-03-04T16:19:16.801 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030c95c5c24787bee-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.804 DEBUG:boto:path=/sbnvju-17/obj18 2024-03-04T16:19:16.804 DEBUG:boto:auth_path=/sbnvju-17/obj18 2024-03-04T16:19:16.804 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.804 DEBUG:boto:Path: /sbnvju-17/obj18 2024-03-04T16:19:16.804 DEBUG:boto:Data: 2024-03-04T16:19:16.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.804 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.805 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.805 DEBUG:boto:Params: {} 2024-03-04T16:19:16.805 DEBUG:boto:Token: None 2024-03-04T16:19:16.805 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj18 2024-03-04T16:19:16.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iJ4hLzKPMVCls4f59Mj5uPXkk38= 2024-03-04T16:19:16.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iJ4hLzKPMVCls4f59Mj5uPXkk38='} 2024-03-04T16:19:16.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2fe476bfc1041b8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.810 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.810 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.810 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.810 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.810 DEBUG:boto:Data: 2024-03-04T16:19:16.810 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.810 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.810 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.810 DEBUG:boto:Params: {} 2024-03-04T16:19:16.810 DEBUG:boto:Token: None 2024-03-04T16:19:16.811 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.811 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099e438a444f6f672-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.813 DEBUG:boto:path=/sbnvju-17/obj19 2024-03-04T16:19:16.813 DEBUG:boto:auth_path=/sbnvju-17/obj19 2024-03-04T16:19:16.813 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.813 DEBUG:boto:Path: /sbnvju-17/obj19 2024-03-04T16:19:16.813 DEBUG:boto:Data: 2024-03-04T16:19:16.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.813 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.813 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.814 DEBUG:boto:Params: {} 2024-03-04T16:19:16.814 DEBUG:boto:Token: None 2024-03-04T16:19:16.814 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj19 2024-03-04T16:19:16.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:97vhgY1JtC6RX8d+FrSYCAjy9ts= 2024-03-04T16:19:16.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:97vhgY1JtC6RX8d+FrSYCAjy9ts='} 2024-03-04T16:19:16.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf2ada074ff841a8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.860 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.861 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.861 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.861 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.861 DEBUG:boto:Data: 2024-03-04T16:19:16.861 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.861 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.861 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.861 DEBUG:boto:Params: {} 2024-03-04T16:19:16.861 DEBUG:boto:Token: None 2024-03-04T16:19:16.861 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003d056763677a5ab-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.864 DEBUG:boto:path=/sbnvju-17/obj20 2024-03-04T16:19:16.864 DEBUG:boto:auth_path=/sbnvju-17/obj20 2024-03-04T16:19:16.864 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.864 DEBUG:boto:Path: /sbnvju-17/obj20 2024-03-04T16:19:16.864 DEBUG:boto:Data: 2024-03-04T16:19:16.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.865 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.865 DEBUG:boto:Params: {} 2024-03-04T16:19:16.865 DEBUG:boto:Token: None 2024-03-04T16:19:16.865 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj20 2024-03-04T16:19:16.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aGj1iS8t4P/Yw3GOAyKn+npwrhw= 2024-03-04T16:19:16.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aGj1iS8t4P/Yw3GOAyKn+npwrhw='} 2024-03-04T16:19:16.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007d1f8c0b4fcb2d0-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.871 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.871 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.871 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.871 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.871 DEBUG:boto:Data: 2024-03-04T16:19:16.871 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.871 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.871 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.871 DEBUG:boto:Params: {} 2024-03-04T16:19:16.871 DEBUG:boto:Token: None 2024-03-04T16:19:16.871 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.872 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c35b797a47a3886-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.874 DEBUG:boto:path=/sbnvju-17/obj21 2024-03-04T16:19:16.874 DEBUG:boto:auth_path=/sbnvju-17/obj21 2024-03-04T16:19:16.874 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.874 DEBUG:boto:Path: /sbnvju-17/obj21 2024-03-04T16:19:16.874 DEBUG:boto:Data: 2024-03-04T16:19:16.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.874 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.875 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.875 DEBUG:boto:Params: {} 2024-03-04T16:19:16.875 DEBUG:boto:Token: None 2024-03-04T16:19:16.875 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj21 2024-03-04T16:19:16.875 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ebig/WLz2F8oTtUXgPn3bpvzetM= 2024-03-04T16:19:16.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ebig/WLz2F8oTtUXgPn3bpvzetM='} 2024-03-04T16:19:16.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000043e5d51a3c7f40f-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.880 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.880 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.881 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.881 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.881 DEBUG:boto:Data: 2024-03-04T16:19:16.881 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.881 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.881 DEBUG:boto:Params: {} 2024-03-04T16:19:16.881 DEBUG:boto:Token: None 2024-03-04T16:19:16.881 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.881 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9bf0ac6d3bf6c2e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.884 DEBUG:boto:path=/sbnvju-17/obj22 2024-03-04T16:19:16.884 DEBUG:boto:auth_path=/sbnvju-17/obj22 2024-03-04T16:19:16.884 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.884 DEBUG:boto:Path: /sbnvju-17/obj22 2024-03-04T16:19:16.884 DEBUG:boto:Data: 2024-03-04T16:19:16.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.884 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.884 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.884 DEBUG:boto:Params: {} 2024-03-04T16:19:16.884 DEBUG:boto:Token: None 2024-03-04T16:19:16.884 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj22 2024-03-04T16:19:16.885 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lltJhKhfYWGVRKhtZuEmG+ZX1l8= 2024-03-04T16:19:16.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lltJhKhfYWGVRKhtZuEmG+ZX1l8='} 2024-03-04T16:19:16.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005161c15c0c3552aa-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.928 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.928 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.928 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.929 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.929 DEBUG:boto:Data: 2024-03-04T16:19:16.929 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.929 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.929 DEBUG:boto:Params: {} 2024-03-04T16:19:16.929 DEBUG:boto:Token: None 2024-03-04T16:19:16.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.929 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a95ce1a07c5114ce-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.932 DEBUG:boto:path=/sbnvju-17/obj23 2024-03-04T16:19:16.932 DEBUG:boto:auth_path=/sbnvju-17/obj23 2024-03-04T16:19:16.932 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.932 DEBUG:boto:Path: /sbnvju-17/obj23 2024-03-04T16:19:16.932 DEBUG:boto:Data: 2024-03-04T16:19:16.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.932 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.932 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.932 DEBUG:boto:Params: {} 2024-03-04T16:19:16.933 DEBUG:boto:Token: None 2024-03-04T16:19:16.933 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj23 2024-03-04T16:19:16.933 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p/kmNz7UOEGdk9XnyGJs5gDgAc4= 2024-03-04T16:19:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p/kmNz7UOEGdk9XnyGJs5gDgAc4='} 2024-03-04T16:19:16.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7991a670ac7b3ff-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.937 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.937 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.937 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.937 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.937 DEBUG:boto:Data: 2024-03-04T16:19:16.938 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.938 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.938 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.938 DEBUG:boto:Params: {} 2024-03-04T16:19:16.938 DEBUG:boto:Token: None 2024-03-04T16:19:16.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000714aa866f3c31f39-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.940 DEBUG:boto:path=/sbnvju-17/obj24 2024-03-04T16:19:16.940 DEBUG:boto:auth_path=/sbnvju-17/obj24 2024-03-04T16:19:16.940 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.940 DEBUG:boto:Path: /sbnvju-17/obj24 2024-03-04T16:19:16.940 DEBUG:boto:Data: 2024-03-04T16:19:16.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.941 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.941 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.941 DEBUG:boto:Params: {} 2024-03-04T16:19:16.941 DEBUG:boto:Token: None 2024-03-04T16:19:16.941 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj24 2024-03-04T16:19:16.941 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wQtDBDZffjRCnhvjhhrpTy0DURg= 2024-03-04T16:19:16.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wQtDBDZffjRCnhvjhhrpTy0DURg='} 2024-03-04T16:19:16.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002581492affd0d132-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.945 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.945 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.945 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.945 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.945 DEBUG:boto:Data: 2024-03-04T16:19:16.945 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.946 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.946 DEBUG:boto:Params: {} 2024-03-04T16:19:16.946 DEBUG:boto:Token: None 2024-03-04T16:19:16.946 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006b0e48e74174534-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.949 DEBUG:boto:path=/sbnvju-17/obj25 2024-03-04T16:19:16.949 DEBUG:boto:auth_path=/sbnvju-17/obj25 2024-03-04T16:19:16.949 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.949 DEBUG:boto:Path: /sbnvju-17/obj25 2024-03-04T16:19:16.949 DEBUG:boto:Data: 2024-03-04T16:19:16.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.949 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.949 DEBUG:boto:Params: {} 2024-03-04T16:19:16.949 DEBUG:boto:Token: None 2024-03-04T16:19:16.950 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj25 2024-03-04T16:19:16.950 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yxRIMEOmM8PI7tI0CfBivaDAMcE= 2024-03-04T16:19:16.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yxRIMEOmM8PI7tI0CfBivaDAMcE='} 2024-03-04T16:19:16.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aac2f1b11e234e0-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.992 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:16.992 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:16.993 DEBUG:boto:Method: HEAD 2024-03-04T16:19:16.993 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:16.993 DEBUG:boto:Data: 2024-03-04T16:19:16.993 DEBUG:boto:Headers: {} 2024-03-04T16:19:16.993 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.993 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.993 DEBUG:boto:Params: {} 2024-03-04T16:19:16.993 DEBUG:boto:Token: None 2024-03-04T16:19:16.993 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/ 2024-03-04T16:19:16.993 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T16:19:16.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T16:19:16.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4bb3a260ce40e26-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:16.996 DEBUG:boto:path=/sbnvju-17/obj26 2024-03-04T16:19:16.996 DEBUG:boto:auth_path=/sbnvju-17/obj26 2024-03-04T16:19:16.996 DEBUG:boto:Method: PUT 2024-03-04T16:19:16.996 DEBUG:boto:Path: /sbnvju-17/obj26 2024-03-04T16:19:16.996 DEBUG:boto:Data: 2024-03-04T16:19:16.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:16.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:16.997 DEBUG:boto:Port: 8000 2024-03-04T16:19:16.997 DEBUG:boto:Params: {} 2024-03-04T16:19:16.997 DEBUG:boto:Token: None 2024-03-04T16:19:16.997 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:16 GMT /sbnvju-17/obj26 2024-03-04T16:19:16.997 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H/DSiX3LLS98AaGgwzYNcym5QDw= 2024-03-04T16:19:16.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H/DSiX3LLS98AaGgwzYNcym5QDw='} 2024-03-04T16:19:17.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e64483029edbe48d-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.001 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.001 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.001 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.001 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.002 DEBUG:boto:Data: 2024-03-04T16:19:17.002 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.002 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.002 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.002 DEBUG:boto:Params: {} 2024-03-04T16:19:17.002 DEBUG:boto:Token: None 2024-03-04T16:19:17.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000001c5b868b090d7-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.004 DEBUG:boto:path=/sbnvju-17/obj27 2024-03-04T16:19:17.005 DEBUG:boto:auth_path=/sbnvju-17/obj27 2024-03-04T16:19:17.005 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.005 DEBUG:boto:Path: /sbnvju-17/obj27 2024-03-04T16:19:17.005 DEBUG:boto:Data: 2024-03-04T16:19:17.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.005 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.005 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.005 DEBUG:boto:Params: {} 2024-03-04T16:19:17.005 DEBUG:boto:Token: None 2024-03-04T16:19:17.005 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj27 2024-03-04T16:19:17.006 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0AR3gJd5La8EgJb8363GB5DNvRg= 2024-03-04T16:19:17.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0AR3gJd5La8EgJb8363GB5DNvRg='} 2024-03-04T16:19:17.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e42acd0bdb5fca3-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.048 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.048 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.049 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.049 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.049 DEBUG:boto:Data: 2024-03-04T16:19:17.049 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.049 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.049 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.049 DEBUG:boto:Params: {} 2024-03-04T16:19:17.049 DEBUG:boto:Token: None 2024-03-04T16:19:17.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d67528dff1d38ea-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.052 DEBUG:boto:path=/sbnvju-17/obj28 2024-03-04T16:19:17.052 DEBUG:boto:auth_path=/sbnvju-17/obj28 2024-03-04T16:19:17.052 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.052 DEBUG:boto:Path: /sbnvju-17/obj28 2024-03-04T16:19:17.052 DEBUG:boto:Data: 2024-03-04T16:19:17.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.052 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.052 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.053 DEBUG:boto:Params: {} 2024-03-04T16:19:17.053 DEBUG:boto:Token: None 2024-03-04T16:19:17.053 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj28 2024-03-04T16:19:17.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zN1xG+5jSwDSFg1RzyHd2519Wpo= 2024-03-04T16:19:17.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zN1xG+5jSwDSFg1RzyHd2519Wpo='} 2024-03-04T16:19:17.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000baf2c453c4a6893b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.057 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.057 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.057 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.057 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.057 DEBUG:boto:Data: 2024-03-04T16:19:17.057 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.057 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.057 DEBUG:boto:Params: {} 2024-03-04T16:19:17.058 DEBUG:boto:Token: None 2024-03-04T16:19:17.058 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063a86f9c7fcf0734-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.060 DEBUG:boto:path=/sbnvju-17/obj29 2024-03-04T16:19:17.060 DEBUG:boto:auth_path=/sbnvju-17/obj29 2024-03-04T16:19:17.060 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.060 DEBUG:boto:Path: /sbnvju-17/obj29 2024-03-04T16:19:17.060 DEBUG:boto:Data: 2024-03-04T16:19:17.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.060 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.060 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.061 DEBUG:boto:Params: {} 2024-03-04T16:19:17.061 DEBUG:boto:Token: None 2024-03-04T16:19:17.061 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj29 2024-03-04T16:19:17.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V8Q25hmdDKY3aUAL2kPwH0QX2Fs= 2024-03-04T16:19:17.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V8Q25hmdDKY3aUAL2kPwH0QX2Fs='} 2024-03-04T16:19:17.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcb4180d3460ab27-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.065 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.065 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.065 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.065 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.065 DEBUG:boto:Data: 2024-03-04T16:19:17.065 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.065 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.065 DEBUG:boto:Params: {} 2024-03-04T16:19:17.065 DEBUG:boto:Token: None 2024-03-04T16:19:17.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8986ef13cd92561-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.068 DEBUG:boto:path=/sbnvju-17/obj30 2024-03-04T16:19:17.068 DEBUG:boto:auth_path=/sbnvju-17/obj30 2024-03-04T16:19:17.068 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.068 DEBUG:boto:Path: /sbnvju-17/obj30 2024-03-04T16:19:17.068 DEBUG:boto:Data: 2024-03-04T16:19:17.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.068 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.069 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.069 DEBUG:boto:Params: {} 2024-03-04T16:19:17.069 DEBUG:boto:Token: None 2024-03-04T16:19:17.069 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj30 2024-03-04T16:19:17.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/BMvJ4No/x0F+cDvT1vdvmg5S7o= 2024-03-04T16:19:17.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/BMvJ4No/x0F+cDvT1vdvmg5S7o='} 2024-03-04T16:19:17.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044dcc1395eb152b4-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.112 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.112 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.113 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.113 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.113 DEBUG:boto:Data: 2024-03-04T16:19:17.113 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.113 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.113 DEBUG:boto:Params: {} 2024-03-04T16:19:17.113 DEBUG:boto:Token: None 2024-03-04T16:19:17.113 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.113 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8d84e8ffebd2001-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.116 DEBUG:boto:path=/sbnvju-17/obj31 2024-03-04T16:19:17.116 DEBUG:boto:auth_path=/sbnvju-17/obj31 2024-03-04T16:19:17.116 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.116 DEBUG:boto:Path: /sbnvju-17/obj31 2024-03-04T16:19:17.116 DEBUG:boto:Data: 2024-03-04T16:19:17.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.116 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.116 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.116 DEBUG:boto:Params: {} 2024-03-04T16:19:17.117 DEBUG:boto:Token: None 2024-03-04T16:19:17.117 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj31 2024-03-04T16:19:17.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jZSgzLakSU35jgRO/zepL0q43oU= 2024-03-04T16:19:17.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jZSgzLakSU35jgRO/zepL0q43oU='} 2024-03-04T16:19:17.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be651e6447cb0709-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.121 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.121 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.121 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.121 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.121 DEBUG:boto:Data: 2024-03-04T16:19:17.121 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.122 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.122 DEBUG:boto:Params: {} 2024-03-04T16:19:17.122 DEBUG:boto:Token: None 2024-03-04T16:19:17.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad87a72313bff84d-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.124 DEBUG:boto:path=/sbnvju-17/obj32 2024-03-04T16:19:17.124 DEBUG:boto:auth_path=/sbnvju-17/obj32 2024-03-04T16:19:17.124 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.124 DEBUG:boto:Path: /sbnvju-17/obj32 2024-03-04T16:19:17.124 DEBUG:boto:Data: 2024-03-04T16:19:17.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.125 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.125 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.125 DEBUG:boto:Params: {} 2024-03-04T16:19:17.125 DEBUG:boto:Token: None 2024-03-04T16:19:17.125 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj32 2024-03-04T16:19:17.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YSBSfrcDNxL7Mn3e0H68oASpdo0= 2024-03-04T16:19:17.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YSBSfrcDNxL7Mn3e0H68oASpdo0='} 2024-03-04T16:19:17.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef52b73001aa4b7e-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.168 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.168 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.169 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.169 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.169 DEBUG:boto:Data: 2024-03-04T16:19:17.169 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.169 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.169 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.169 DEBUG:boto:Params: {} 2024-03-04T16:19:17.169 DEBUG:boto:Token: None 2024-03-04T16:19:17.169 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d02583d65d52457f-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.172 DEBUG:boto:path=/sbnvju-17/obj33 2024-03-04T16:19:17.172 DEBUG:boto:auth_path=/sbnvju-17/obj33 2024-03-04T16:19:17.172 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.172 DEBUG:boto:Path: /sbnvju-17/obj33 2024-03-04T16:19:17.172 DEBUG:boto:Data: 2024-03-04T16:19:17.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.172 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.173 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.173 DEBUG:boto:Params: {} 2024-03-04T16:19:17.173 DEBUG:boto:Token: None 2024-03-04T16:19:17.173 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj33 2024-03-04T16:19:17.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mW8NaBfjLJEzokuTy5BIQUXoBBQ= 2024-03-04T16:19:17.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mW8NaBfjLJEzokuTy5BIQUXoBBQ='} 2024-03-04T16:19:17.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056be2cafe3d85a72-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.177 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.177 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.177 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.178 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.178 DEBUG:boto:Data: 2024-03-04T16:19:17.178 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.178 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.178 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.178 DEBUG:boto:Params: {} 2024-03-04T16:19:17.178 DEBUG:boto:Token: None 2024-03-04T16:19:17.178 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000138c1e0959dfe0f2-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.180 DEBUG:boto:path=/sbnvju-17/obj34 2024-03-04T16:19:17.180 DEBUG:boto:auth_path=/sbnvju-17/obj34 2024-03-04T16:19:17.181 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.181 DEBUG:boto:Path: /sbnvju-17/obj34 2024-03-04T16:19:17.181 DEBUG:boto:Data: 2024-03-04T16:19:17.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.181 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.181 DEBUG:boto:Params: {} 2024-03-04T16:19:17.181 DEBUG:boto:Token: None 2024-03-04T16:19:17.181 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj34 2024-03-04T16:19:17.181 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I1cvtrlhxBFXpynbljfQHPKVh2s= 2024-03-04T16:19:17.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I1cvtrlhxBFXpynbljfQHPKVh2s='} 2024-03-04T16:19:17.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c3a50cde03720ed-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.228 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.228 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.228 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.229 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.229 DEBUG:boto:Data: 2024-03-04T16:19:17.229 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.229 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.229 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.229 DEBUG:boto:Params: {} 2024-03-04T16:19:17.229 DEBUG:boto:Token: None 2024-03-04T16:19:17.229 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ced6eec3c6629b8-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.231 DEBUG:boto:path=/sbnvju-17/obj35 2024-03-04T16:19:17.232 DEBUG:boto:auth_path=/sbnvju-17/obj35 2024-03-04T16:19:17.232 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.232 DEBUG:boto:Path: /sbnvju-17/obj35 2024-03-04T16:19:17.232 DEBUG:boto:Data: 2024-03-04T16:19:17.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.232 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.232 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.232 DEBUG:boto:Params: {} 2024-03-04T16:19:17.232 DEBUG:boto:Token: None 2024-03-04T16:19:17.232 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj35 2024-03-04T16:19:17.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rclb1FkpEEErsRlyhFt1V0rv+e4= 2024-03-04T16:19:17.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rclb1FkpEEErsRlyhFt1V0rv+e4='} 2024-03-04T16:19:17.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087b5a223ac7b5f81-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.236 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.236 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.236 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.236 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.237 DEBUG:boto:Data: 2024-03-04T16:19:17.237 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.237 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.237 DEBUG:boto:Params: {} 2024-03-04T16:19:17.237 DEBUG:boto:Token: None 2024-03-04T16:19:17.237 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c6e7b1e80fcbf6-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.240 DEBUG:boto:path=/sbnvju-17/obj36 2024-03-04T16:19:17.240 DEBUG:boto:auth_path=/sbnvju-17/obj36 2024-03-04T16:19:17.240 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.240 DEBUG:boto:Path: /sbnvju-17/obj36 2024-03-04T16:19:17.240 DEBUG:boto:Data: 2024-03-04T16:19:17.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.240 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.240 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.240 DEBUG:boto:Params: {} 2024-03-04T16:19:17.241 DEBUG:boto:Token: None 2024-03-04T16:19:17.241 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj36 2024-03-04T16:19:17.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RFmEdUNrvAmIkPL75xpVX2B8tjI= 2024-03-04T16:19:17.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RFmEdUNrvAmIkPL75xpVX2B8tjI='} 2024-03-04T16:19:17.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000896358968d0ef778-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.244 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.245 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.245 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.245 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.245 DEBUG:boto:Data: 2024-03-04T16:19:17.245 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.245 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.245 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.245 DEBUG:boto:Params: {} 2024-03-04T16:19:17.245 DEBUG:boto:Token: None 2024-03-04T16:19:17.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a9f3db3ae9c5595-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.247 DEBUG:boto:path=/sbnvju-17/obj37 2024-03-04T16:19:17.248 DEBUG:boto:auth_path=/sbnvju-17/obj37 2024-03-04T16:19:17.248 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.248 DEBUG:boto:Path: /sbnvju-17/obj37 2024-03-04T16:19:17.248 DEBUG:boto:Data: 2024-03-04T16:19:17.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.248 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.248 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.248 DEBUG:boto:Params: {} 2024-03-04T16:19:17.248 DEBUG:boto:Token: None 2024-03-04T16:19:17.248 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj37 2024-03-04T16:19:17.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wyfrEzimDpGgPblxzVP/we1rBPA= 2024-03-04T16:19:17.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wyfrEzimDpGgPblxzVP/we1rBPA='} 2024-03-04T16:19:17.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bfaecca3db5d973-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.296 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.297 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.297 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.297 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.297 DEBUG:boto:Data: 2024-03-04T16:19:17.297 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.297 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.297 DEBUG:boto:Params: {} 2024-03-04T16:19:17.297 DEBUG:boto:Token: None 2024-03-04T16:19:17.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec0c0ac6e502b6c8-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.300 DEBUG:boto:path=/sbnvju-17/obj38 2024-03-04T16:19:17.300 DEBUG:boto:auth_path=/sbnvju-17/obj38 2024-03-04T16:19:17.300 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.300 DEBUG:boto:Path: /sbnvju-17/obj38 2024-03-04T16:19:17.300 DEBUG:boto:Data: 2024-03-04T16:19:17.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.300 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.300 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.301 DEBUG:boto:Params: {} 2024-03-04T16:19:17.301 DEBUG:boto:Token: None 2024-03-04T16:19:17.301 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj38 2024-03-04T16:19:17.301 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OUxhfBWuNHc1FwcBkWhSCgaisMw= 2024-03-04T16:19:17.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OUxhfBWuNHc1FwcBkWhSCgaisMw='} 2024-03-04T16:19:17.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4003625a3c8a3c1-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.304 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.305 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.305 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.305 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.305 DEBUG:boto:Data: 2024-03-04T16:19:17.305 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.305 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.305 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.305 DEBUG:boto:Params: {} 2024-03-04T16:19:17.305 DEBUG:boto:Token: None 2024-03-04T16:19:17.306 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee5241f9fb8dc6db-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.308 DEBUG:boto:path=/sbnvju-17/obj39 2024-03-04T16:19:17.308 DEBUG:boto:auth_path=/sbnvju-17/obj39 2024-03-04T16:19:17.308 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.308 DEBUG:boto:Path: /sbnvju-17/obj39 2024-03-04T16:19:17.308 DEBUG:boto:Data: 2024-03-04T16:19:17.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.308 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.308 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.308 DEBUG:boto:Params: {} 2024-03-04T16:19:17.309 DEBUG:boto:Token: None 2024-03-04T16:19:17.309 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj39 2024-03-04T16:19:17.309 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TZzCoUBDvGIUh3oEJpNZ7BJIAk8= 2024-03-04T16:19:17.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TZzCoUBDvGIUh3oEJpNZ7BJIAk8='} 2024-03-04T16:19:17.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c351632b1b596dc-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.315 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.315 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.316 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.316 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.316 DEBUG:boto:Data: 2024-03-04T16:19:17.316 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.316 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.316 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.316 DEBUG:boto:Params: {} 2024-03-04T16:19:17.316 DEBUG:boto:Token: None 2024-03-04T16:19:17.316 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.317 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b64bd0d47dc9e31a-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.319 DEBUG:boto:path=/sbnvju-17/obj40 2024-03-04T16:19:17.319 DEBUG:boto:auth_path=/sbnvju-17/obj40 2024-03-04T16:19:17.319 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.319 DEBUG:boto:Path: /sbnvju-17/obj40 2024-03-04T16:19:17.319 DEBUG:boto:Data: 2024-03-04T16:19:17.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.319 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.319 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.320 DEBUG:boto:Params: {} 2024-03-04T16:19:17.320 DEBUG:boto:Token: None 2024-03-04T16:19:17.320 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj40 2024-03-04T16:19:17.320 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u0vA9Shxju3NmKA//vh4ovsas+w= 2024-03-04T16:19:17.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u0vA9Shxju3NmKA//vh4ovsas+w='} 2024-03-04T16:19:17.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a12e14c42bf4ff2c-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.364 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.364 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.364 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.365 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.365 DEBUG:boto:Data: 2024-03-04T16:19:17.365 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.365 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.365 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.365 DEBUG:boto:Params: {} 2024-03-04T16:19:17.365 DEBUG:boto:Token: None 2024-03-04T16:19:17.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d011d68d46090ef-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.367 DEBUG:boto:path=/sbnvju-17/obj41 2024-03-04T16:19:17.368 DEBUG:boto:auth_path=/sbnvju-17/obj41 2024-03-04T16:19:17.368 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.368 DEBUG:boto:Path: /sbnvju-17/obj41 2024-03-04T16:19:17.368 DEBUG:boto:Data: 2024-03-04T16:19:17.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.368 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.368 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.368 DEBUG:boto:Params: {} 2024-03-04T16:19:17.368 DEBUG:boto:Token: None 2024-03-04T16:19:17.369 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj41 2024-03-04T16:19:17.369 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:U6a96BYyFLeezg82rSUBPHtgcRw= 2024-03-04T16:19:17.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:U6a96BYyFLeezg82rSUBPHtgcRw='} 2024-03-04T16:19:17.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba46b50807163892-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.373 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.373 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.373 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.373 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.373 DEBUG:boto:Data: 2024-03-04T16:19:17.373 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.373 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.373 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.373 DEBUG:boto:Params: {} 2024-03-04T16:19:17.373 DEBUG:boto:Token: None 2024-03-04T16:19:17.374 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.374 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a66384b605b03678-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.376 DEBUG:boto:path=/sbnvju-17/obj42 2024-03-04T16:19:17.376 DEBUG:boto:auth_path=/sbnvju-17/obj42 2024-03-04T16:19:17.376 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.376 DEBUG:boto:Path: /sbnvju-17/obj42 2024-03-04T16:19:17.376 DEBUG:boto:Data: 2024-03-04T16:19:17.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.377 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.377 DEBUG:boto:Params: {} 2024-03-04T16:19:17.377 DEBUG:boto:Token: None 2024-03-04T16:19:17.377 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj42 2024-03-04T16:19:17.377 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2j5oWPieZi03g+FUweG96vrzq94= 2024-03-04T16:19:17.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2j5oWPieZi03g+FUweG96vrzq94='} 2024-03-04T16:19:17.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008495ee0f0c2da30-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.381 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.381 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.381 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.381 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.381 DEBUG:boto:Data: 2024-03-04T16:19:17.381 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.381 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.381 DEBUG:boto:Params: {} 2024-03-04T16:19:17.382 DEBUG:boto:Token: None 2024-03-04T16:19:17.382 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc8a7ed29f12e7b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.384 DEBUG:boto:path=/sbnvju-17/obj43 2024-03-04T16:19:17.384 DEBUG:boto:auth_path=/sbnvju-17/obj43 2024-03-04T16:19:17.384 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.384 DEBUG:boto:Path: /sbnvju-17/obj43 2024-03-04T16:19:17.384 DEBUG:boto:Data: 2024-03-04T16:19:17.385 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.385 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.385 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.385 DEBUG:boto:Params: {} 2024-03-04T16:19:17.385 DEBUG:boto:Token: None 2024-03-04T16:19:17.385 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj43 2024-03-04T16:19:17.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:deDPa/lZBHHpGGihPNY6tVGK+f8= 2024-03-04T16:19:17.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:deDPa/lZBHHpGGihPNY6tVGK+f8='} 2024-03-04T16:19:17.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cf8d53a950ae15d-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.432 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.432 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.433 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.433 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.433 DEBUG:boto:Data: 2024-03-04T16:19:17.433 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.433 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.433 DEBUG:boto:Params: {} 2024-03-04T16:19:17.433 DEBUG:boto:Token: None 2024-03-04T16:19:17.433 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2a0421c242a3481-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.436 DEBUG:boto:path=/sbnvju-17/obj44 2024-03-04T16:19:17.436 DEBUG:boto:auth_path=/sbnvju-17/obj44 2024-03-04T16:19:17.436 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.436 DEBUG:boto:Path: /sbnvju-17/obj44 2024-03-04T16:19:17.436 DEBUG:boto:Data: 2024-03-04T16:19:17.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.437 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.437 DEBUG:boto:Params: {} 2024-03-04T16:19:17.437 DEBUG:boto:Token: None 2024-03-04T16:19:17.437 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj44 2024-03-04T16:19:17.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M4dWY5O+xF7Qi/4qLG/XwtDMEw4= 2024-03-04T16:19:17.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M4dWY5O+xF7Qi/4qLG/XwtDMEw4='} 2024-03-04T16:19:17.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcfac8c6753e6be9-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.442 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.442 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.442 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.442 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.443 DEBUG:boto:Data: 2024-03-04T16:19:17.443 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.443 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.443 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.443 DEBUG:boto:Params: {} 2024-03-04T16:19:17.443 DEBUG:boto:Token: None 2024-03-04T16:19:17.443 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002da2624d493e835c-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.445 DEBUG:boto:path=/sbnvju-17/obj45 2024-03-04T16:19:17.445 DEBUG:boto:auth_path=/sbnvju-17/obj45 2024-03-04T16:19:17.446 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.446 DEBUG:boto:Path: /sbnvju-17/obj45 2024-03-04T16:19:17.446 DEBUG:boto:Data: 2024-03-04T16:19:17.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.446 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.446 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.446 DEBUG:boto:Params: {} 2024-03-04T16:19:17.446 DEBUG:boto:Token: None 2024-03-04T16:19:17.446 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj45 2024-03-04T16:19:17.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dJS2MMTY8+FwxEvRYqOo9217Irs= 2024-03-04T16:19:17.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dJS2MMTY8+FwxEvRYqOo9217Irs='} 2024-03-04T16:19:17.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000747698855a5626cb-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.488 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.489 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.489 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.489 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.489 DEBUG:boto:Data: 2024-03-04T16:19:17.489 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.489 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.489 DEBUG:boto:Params: {} 2024-03-04T16:19:17.489 DEBUG:boto:Token: None 2024-03-04T16:19:17.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.490 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5bafa6616314aab-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.492 DEBUG:boto:path=/sbnvju-17/obj46 2024-03-04T16:19:17.492 DEBUG:boto:auth_path=/sbnvju-17/obj46 2024-03-04T16:19:17.492 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.492 DEBUG:boto:Path: /sbnvju-17/obj46 2024-03-04T16:19:17.493 DEBUG:boto:Data: 2024-03-04T16:19:17.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.493 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.493 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.493 DEBUG:boto:Params: {} 2024-03-04T16:19:17.493 DEBUG:boto:Token: None 2024-03-04T16:19:17.493 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj46 2024-03-04T16:19:17.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ovMiGcDLoH0XRAE4+M5DUhV40M= 2024-03-04T16:19:17.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ovMiGcDLoH0XRAE4+M5DUhV40M='} 2024-03-04T16:19:17.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3481f693aeed615-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.497 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.497 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.497 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.497 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.497 DEBUG:boto:Data: 2024-03-04T16:19:17.498 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.498 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.498 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.498 DEBUG:boto:Params: {} 2024-03-04T16:19:17.498 DEBUG:boto:Token: None 2024-03-04T16:19:17.498 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.498 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f139df0a57e0a21b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.500 DEBUG:boto:path=/sbnvju-17/obj47 2024-03-04T16:19:17.500 DEBUG:boto:auth_path=/sbnvju-17/obj47 2024-03-04T16:19:17.501 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.501 DEBUG:boto:Path: /sbnvju-17/obj47 2024-03-04T16:19:17.501 DEBUG:boto:Data: 2024-03-04T16:19:17.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.501 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.501 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.501 DEBUG:boto:Params: {} 2024-03-04T16:19:17.501 DEBUG:boto:Token: None 2024-03-04T16:19:17.501 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj47 2024-03-04T16:19:17.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2xBs1QnztYesm+5UKRLmi+8L7bo= 2024-03-04T16:19:17.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2xBs1QnztYesm+5UKRLmi+8L7bo='} 2024-03-04T16:19:17.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062bacfcb22569f31-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.544 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.544 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.545 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.545 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.545 DEBUG:boto:Data: 2024-03-04T16:19:17.545 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.545 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.545 DEBUG:boto:Params: {} 2024-03-04T16:19:17.545 DEBUG:boto:Token: None 2024-03-04T16:19:17.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f32c711808bbc481-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.548 DEBUG:boto:path=/sbnvju-17/obj48 2024-03-04T16:19:17.548 DEBUG:boto:auth_path=/sbnvju-17/obj48 2024-03-04T16:19:17.548 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.548 DEBUG:boto:Path: /sbnvju-17/obj48 2024-03-04T16:19:17.548 DEBUG:boto:Data: 2024-03-04T16:19:17.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.548 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.549 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.549 DEBUG:boto:Params: {} 2024-03-04T16:19:17.549 DEBUG:boto:Token: None 2024-03-04T16:19:17.549 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj48 2024-03-04T16:19:17.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TO1IBJsjDvI47LwQOxMkrCESFS4= 2024-03-04T16:19:17.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TO1IBJsjDvI47LwQOxMkrCESFS4='} 2024-03-04T16:19:17.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a52c90e18d1fd59-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.553 DEBUG:boto:path=/sbnvju-17/ 2024-03-04T16:19:17.553 DEBUG:boto:auth_path=/sbnvju-17/ 2024-03-04T16:19:17.553 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.553 DEBUG:boto:Path: /sbnvju-17/ 2024-03-04T16:19:17.553 DEBUG:boto:Data: 2024-03-04T16:19:17.553 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.553 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.553 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.554 DEBUG:boto:Params: {} 2024-03-04T16:19:17.554 DEBUG:boto:Token: None 2024-03-04T16:19:17.554 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/ 2024-03-04T16:19:17.554 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T16:19:17.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T16:19:17.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048375c65b449f682-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.556 DEBUG:boto:path=/sbnvju-17/obj49 2024-03-04T16:19:17.556 DEBUG:boto:auth_path=/sbnvju-17/obj49 2024-03-04T16:19:17.556 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.556 DEBUG:boto:Path: /sbnvju-17/obj49 2024-03-04T16:19:17.557 DEBUG:boto:Data: 2024-03-04T16:19:17.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.557 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:19:17.557 DEBUG:boto:Port: 8000 2024-03-04T16:19:17.557 DEBUG:boto:Params: {} 2024-03-04T16:19:17.557 DEBUG:boto:Token: None 2024-03-04T16:19:17.557 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-17/obj49 2024-03-04T16:19:17.557 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LjANaMIz/YXH+MURHZHBq60BvP4= 2024-03-04T16:19:17.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LjANaMIz/YXH+MURHZHBq60BvP4='} 2024-03-04T16:19:17.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090435843a9a46d87-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.561 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.561 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.561 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.562 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.562 DEBUG:boto:Data: 2024-03-04T16:19:17.562 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.562 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.562 DEBUG:boto:Params: {} 2024-03-04T16:19:17.562 DEBUG:boto:Token: None 2024-03-04T16:19:17.562 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f39c0e73469b21a-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.565 DEBUG:boto:path=/sbnvju-18/obj1 2024-03-04T16:19:17.565 DEBUG:boto:auth_path=/sbnvju-18/obj1 2024-03-04T16:19:17.565 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.565 DEBUG:boto:Path: /sbnvju-18/obj1 2024-03-04T16:19:17.565 DEBUG:boto:Data: 2024-03-04T16:19:17.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.566 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.566 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.566 DEBUG:boto:Params: {} 2024-03-04T16:19:17.566 DEBUG:boto:Token: None 2024-03-04T16:19:17.566 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj1 2024-03-04T16:19:17.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vz6rEU6rvodg2jePO8NMBGen/0E= 2024-03-04T16:19:17.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vz6rEU6rvodg2jePO8NMBGen/0E='} 2024-03-04T16:19:17.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000241a5d08ed3ae648-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.574 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.574 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.574 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.574 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.574 DEBUG:boto:Data: 2024-03-04T16:19:17.574 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.574 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.574 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.575 DEBUG:boto:Params: {} 2024-03-04T16:19:17.575 DEBUG:boto:Token: None 2024-03-04T16:19:17.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a30beeba3c9ee432-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.578 DEBUG:boto:path=/sbnvju-18/obj2 2024-03-04T16:19:17.578 DEBUG:boto:auth_path=/sbnvju-18/obj2 2024-03-04T16:19:17.578 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.578 DEBUG:boto:Path: /sbnvju-18/obj2 2024-03-04T16:19:17.578 DEBUG:boto:Data: 2024-03-04T16:19:17.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.578 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.578 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.578 DEBUG:boto:Params: {} 2024-03-04T16:19:17.578 DEBUG:boto:Token: None 2024-03-04T16:19:17.579 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj2 2024-03-04T16:19:17.579 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o7/GzzWcApVcZtKXFG5TRIv+kLc= 2024-03-04T16:19:17.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o7/GzzWcApVcZtKXFG5TRIv+kLc='} 2024-03-04T16:19:17.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051012e5d85129936-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.624 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.624 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.625 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.625 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.625 DEBUG:boto:Data: 2024-03-04T16:19:17.625 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.625 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.625 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.625 DEBUG:boto:Params: {} 2024-03-04T16:19:17.625 DEBUG:boto:Token: None 2024-03-04T16:19:17.625 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5452ad68aed90c0-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.628 DEBUG:boto:path=/sbnvju-18/obj3 2024-03-04T16:19:17.628 DEBUG:boto:auth_path=/sbnvju-18/obj3 2024-03-04T16:19:17.628 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.628 DEBUG:boto:Path: /sbnvju-18/obj3 2024-03-04T16:19:17.628 DEBUG:boto:Data: 2024-03-04T16:19:17.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.629 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.629 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.629 DEBUG:boto:Params: {} 2024-03-04T16:19:17.629 DEBUG:boto:Token: None 2024-03-04T16:19:17.629 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj3 2024-03-04T16:19:17.629 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FtoNZppBEl5UBRD1cKljlynW9JQ= 2024-03-04T16:19:17.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FtoNZppBEl5UBRD1cKljlynW9JQ='} 2024-03-04T16:19:17.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5c4c5bebc14add9-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.635 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.636 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.636 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.636 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.636 DEBUG:boto:Data: 2024-03-04T16:19:17.636 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.636 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.636 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.636 DEBUG:boto:Params: {} 2024-03-04T16:19:17.636 DEBUG:boto:Token: None 2024-03-04T16:19:17.636 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.637 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007efad18f8dbf0e1e-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.639 DEBUG:boto:path=/sbnvju-18/obj4 2024-03-04T16:19:17.639 DEBUG:boto:auth_path=/sbnvju-18/obj4 2024-03-04T16:19:17.639 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.640 DEBUG:boto:Path: /sbnvju-18/obj4 2024-03-04T16:19:17.640 DEBUG:boto:Data: 2024-03-04T16:19:17.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.640 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.640 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.640 DEBUG:boto:Params: {} 2024-03-04T16:19:17.640 DEBUG:boto:Token: None 2024-03-04T16:19:17.640 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj4 2024-03-04T16:19:17.640 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rEq/MxCkztIwztZ2DsfU+20WT54= 2024-03-04T16:19:17.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rEq/MxCkztIwztZ2DsfU+20WT54='} 2024-03-04T16:19:17.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efee830ae8044a6c-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.646 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.646 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.646 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.647 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.647 DEBUG:boto:Data: 2024-03-04T16:19:17.647 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.647 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.647 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.647 DEBUG:boto:Params: {} 2024-03-04T16:19:17.647 DEBUG:boto:Token: None 2024-03-04T16:19:17.647 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.647 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc71e69e41f39a65-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.650 DEBUG:boto:path=/sbnvju-18/obj5 2024-03-04T16:19:17.650 DEBUG:boto:auth_path=/sbnvju-18/obj5 2024-03-04T16:19:17.650 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.650 DEBUG:boto:Path: /sbnvju-18/obj5 2024-03-04T16:19:17.650 DEBUG:boto:Data: 2024-03-04T16:19:17.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.650 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.651 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.651 DEBUG:boto:Params: {} 2024-03-04T16:19:17.651 DEBUG:boto:Token: None 2024-03-04T16:19:17.651 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj5 2024-03-04T16:19:17.651 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5gNy/bkx9MlMuGOCypkv6y+ThCk= 2024-03-04T16:19:17.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5gNy/bkx9MlMuGOCypkv6y+ThCk='} 2024-03-04T16:19:17.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c12f1f5774abeaca-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.787 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.787 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.787 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.788 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.788 DEBUG:boto:Data: 2024-03-04T16:19:17.788 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.788 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.788 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.788 DEBUG:boto:Params: {} 2024-03-04T16:19:17.788 DEBUG:boto:Token: None 2024-03-04T16:19:17.788 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002acc2281d4aaed20-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.791 DEBUG:boto:path=/sbnvju-18/obj6 2024-03-04T16:19:17.792 DEBUG:boto:auth_path=/sbnvju-18/obj6 2024-03-04T16:19:17.792 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.792 DEBUG:boto:Path: /sbnvju-18/obj6 2024-03-04T16:19:17.792 DEBUG:boto:Data: 2024-03-04T16:19:17.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.792 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.792 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.792 DEBUG:boto:Params: {} 2024-03-04T16:19:17.792 DEBUG:boto:Token: None 2024-03-04T16:19:17.793 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj6 2024-03-04T16:19:17.793 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gQ2j60OEFqcNKV6ehf1KLupIiH0= 2024-03-04T16:19:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gQ2j60OEFqcNKV6ehf1KLupIiH0='} 2024-03-04T16:19:17.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b91f58890d607dad-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.797 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.797 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.797 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.798 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.798 DEBUG:boto:Data: 2024-03-04T16:19:17.798 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.798 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.798 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.798 DEBUG:boto:Params: {} 2024-03-04T16:19:17.798 DEBUG:boto:Token: None 2024-03-04T16:19:17.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f17be6ac34de7b8d-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.801 DEBUG:boto:path=/sbnvju-18/obj7 2024-03-04T16:19:17.801 DEBUG:boto:auth_path=/sbnvju-18/obj7 2024-03-04T16:19:17.801 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.801 DEBUG:boto:Path: /sbnvju-18/obj7 2024-03-04T16:19:17.801 DEBUG:boto:Data: 2024-03-04T16:19:17.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.801 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.801 DEBUG:boto:Params: {} 2024-03-04T16:19:17.802 DEBUG:boto:Token: None 2024-03-04T16:19:17.802 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj7 2024-03-04T16:19:17.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/+Ioua+H3sX/ik3dOUtRIlFIiBo= 2024-03-04T16:19:17.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/+Ioua+H3sX/ik3dOUtRIlFIiBo='} 2024-03-04T16:19:17.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003908bee176354f7b-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.807 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.807 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.807 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.807 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.807 DEBUG:boto:Data: 2024-03-04T16:19:17.807 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.807 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.807 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.807 DEBUG:boto:Params: {} 2024-03-04T16:19:17.807 DEBUG:boto:Token: None 2024-03-04T16:19:17.808 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.808 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008070562c1c93bc01-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.810 DEBUG:boto:path=/sbnvju-18/obj8 2024-03-04T16:19:17.810 DEBUG:boto:auth_path=/sbnvju-18/obj8 2024-03-04T16:19:17.810 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.811 DEBUG:boto:Path: /sbnvju-18/obj8 2024-03-04T16:19:17.811 DEBUG:boto:Data: 2024-03-04T16:19:17.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.811 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.811 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.811 DEBUG:boto:Params: {} 2024-03-04T16:19:17.811 DEBUG:boto:Token: None 2024-03-04T16:19:17.811 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj8 2024-03-04T16:19:17.811 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tYpyCK+ckbMgcPwo19fA4noOp3w= 2024-03-04T16:19:17.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tYpyCK+ckbMgcPwo19fA4noOp3w='} 2024-03-04T16:19:17.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d52f919f774821c4-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.856 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.856 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.856 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.857 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.857 DEBUG:boto:Data: 2024-03-04T16:19:17.857 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.857 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.857 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.857 DEBUG:boto:Params: {} 2024-03-04T16:19:17.857 DEBUG:boto:Token: None 2024-03-04T16:19:17.857 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b91356aa657f5d-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.860 DEBUG:boto:path=/sbnvju-18/obj9 2024-03-04T16:19:17.860 DEBUG:boto:auth_path=/sbnvju-18/obj9 2024-03-04T16:19:17.860 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.860 DEBUG:boto:Path: /sbnvju-18/obj9 2024-03-04T16:19:17.860 DEBUG:boto:Data: 2024-03-04T16:19:17.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.860 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.860 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.861 DEBUG:boto:Params: {} 2024-03-04T16:19:17.861 DEBUG:boto:Token: None 2024-03-04T16:19:17.861 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj9 2024-03-04T16:19:17.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6JFxLPhZxvSraE/pKz33OHzHOiQ= 2024-03-04T16:19:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6JFxLPhZxvSraE/pKz33OHzHOiQ='} 2024-03-04T16:19:17.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f9a5179da4ec665-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.871 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.871 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.871 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.871 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.871 DEBUG:boto:Data: 2024-03-04T16:19:17.871 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.871 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.871 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.871 DEBUG:boto:Params: {} 2024-03-04T16:19:17.871 DEBUG:boto:Token: None 2024-03-04T16:19:17.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.872 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015a227e1bb48852b-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.877 DEBUG:boto:path=/sbnvju-18/obj10 2024-03-04T16:19:17.877 DEBUG:boto:auth_path=/sbnvju-18/obj10 2024-03-04T16:19:17.877 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.877 DEBUG:boto:Path: /sbnvju-18/obj10 2024-03-04T16:19:17.877 DEBUG:boto:Data: 2024-03-04T16:19:17.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.877 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.877 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.878 DEBUG:boto:Params: {} 2024-03-04T16:19:17.878 DEBUG:boto:Token: None 2024-03-04T16:19:17.878 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj10 2024-03-04T16:19:17.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sE4fnbUt5OoSqs43KWs7NvF94Fw= 2024-03-04T16:19:17.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sE4fnbUt5OoSqs43KWs7NvF94Fw='} 2024-03-04T16:19:17.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008521bc343f115721-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.920 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.920 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.920 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.921 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.921 DEBUG:boto:Data: 2024-03-04T16:19:17.921 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.921 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.921 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.921 DEBUG:boto:Params: {} 2024-03-04T16:19:17.921 DEBUG:boto:Token: None 2024-03-04T16:19:17.921 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056c2dcfacb69b125-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.925 DEBUG:boto:path=/sbnvju-18/obj11 2024-03-04T16:19:17.925 DEBUG:boto:auth_path=/sbnvju-18/obj11 2024-03-04T16:19:17.925 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.925 DEBUG:boto:Path: /sbnvju-18/obj11 2024-03-04T16:19:17.926 DEBUG:boto:Data: 2024-03-04T16:19:17.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.926 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.926 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.926 DEBUG:boto:Params: {} 2024-03-04T16:19:17.926 DEBUG:boto:Token: None 2024-03-04T16:19:17.926 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj11 2024-03-04T16:19:17.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xnnEGhGC9NDYdxuyTFlyADSRbLk= 2024-03-04T16:19:17.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xnnEGhGC9NDYdxuyTFlyADSRbLk='} 2024-03-04T16:19:17.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b516cbdd82ec0d77-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.934 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.935 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.935 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.935 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.935 DEBUG:boto:Data: 2024-03-04T16:19:17.935 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.935 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.935 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.935 DEBUG:boto:Params: {} 2024-03-04T16:19:17.935 DEBUG:boto:Token: None 2024-03-04T16:19:17.935 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.936 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000233177e1fca5f4b9-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.938 DEBUG:boto:path=/sbnvju-18/obj12 2024-03-04T16:19:17.938 DEBUG:boto:auth_path=/sbnvju-18/obj12 2024-03-04T16:19:17.938 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.938 DEBUG:boto:Path: /sbnvju-18/obj12 2024-03-04T16:19:17.938 DEBUG:boto:Data: 2024-03-04T16:19:17.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.938 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.939 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.939 DEBUG:boto:Params: {} 2024-03-04T16:19:17.939 DEBUG:boto:Token: None 2024-03-04T16:19:17.939 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj12 2024-03-04T16:19:17.939 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wRgEhuH5D983BZmT1tIEXZ674eE= 2024-03-04T16:19:17.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wRgEhuH5D983BZmT1tIEXZ674eE='} 2024-03-04T16:19:17.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb648211b99feeac-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.984 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:17.984 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:17.984 DEBUG:boto:Method: HEAD 2024-03-04T16:19:17.985 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:17.985 DEBUG:boto:Data: 2024-03-04T16:19:17.985 DEBUG:boto:Headers: {} 2024-03-04T16:19:17.985 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.985 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.985 DEBUG:boto:Params: {} 2024-03-04T16:19:17.985 DEBUG:boto:Token: None 2024-03-04T16:19:17.985 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/ 2024-03-04T16:19:17.985 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T16:19:17.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T16:19:17.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb17969e06e826f3-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:17.988 DEBUG:boto:path=/sbnvju-18/obj13 2024-03-04T16:19:17.988 DEBUG:boto:auth_path=/sbnvju-18/obj13 2024-03-04T16:19:17.988 DEBUG:boto:Method: PUT 2024-03-04T16:19:17.988 DEBUG:boto:Path: /sbnvju-18/obj13 2024-03-04T16:19:17.988 DEBUG:boto:Data: 2024-03-04T16:19:17.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:17.988 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:17.988 DEBUG:boto:Port: 8001 2024-03-04T16:19:17.988 DEBUG:boto:Params: {} 2024-03-04T16:19:17.989 DEBUG:boto:Token: None 2024-03-04T16:19:17.989 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:17 GMT /sbnvju-18/obj13 2024-03-04T16:19:17.989 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:99h1ak2SH2JgPhEOjjkPk9yLMVo= 2024-03-04T16:19:17.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:99h1ak2SH2JgPhEOjjkPk9yLMVo='} 2024-03-04T16:19:18.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c763682f8118c2e-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.001 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.001 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.001 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.001 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.001 DEBUG:boto:Data: 2024-03-04T16:19:18.001 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.001 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.001 DEBUG:boto:Params: {} 2024-03-04T16:19:18.001 DEBUG:boto:Token: None 2024-03-04T16:19:18.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000755eb979664c8a41-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.006 DEBUG:boto:path=/sbnvju-18/obj14 2024-03-04T16:19:18.006 DEBUG:boto:auth_path=/sbnvju-18/obj14 2024-03-04T16:19:18.006 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.006 DEBUG:boto:Path: /sbnvju-18/obj14 2024-03-04T16:19:18.007 DEBUG:boto:Data: 2024-03-04T16:19:18.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.007 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.007 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.007 DEBUG:boto:Params: {} 2024-03-04T16:19:18.007 DEBUG:boto:Token: None 2024-03-04T16:19:18.007 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj14 2024-03-04T16:19:18.007 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MvFLcqj8W/c0lK60uMcqnzsoSzM= 2024-03-04T16:19:18.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MvFLcqj8W/c0lK60uMcqnzsoSzM='} 2024-03-04T16:19:18.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f664d8a0d40bbce-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.052 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.052 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.052 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.053 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.053 DEBUG:boto:Data: 2024-03-04T16:19:18.053 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.053 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.053 DEBUG:boto:Params: {} 2024-03-04T16:19:18.053 DEBUG:boto:Token: None 2024-03-04T16:19:18.053 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000481709a52099893c-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.056 DEBUG:boto:path=/sbnvju-18/obj15 2024-03-04T16:19:18.056 DEBUG:boto:auth_path=/sbnvju-18/obj15 2024-03-04T16:19:18.056 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.056 DEBUG:boto:Path: /sbnvju-18/obj15 2024-03-04T16:19:18.056 DEBUG:boto:Data: 2024-03-04T16:19:18.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.056 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.057 DEBUG:boto:Params: {} 2024-03-04T16:19:18.057 DEBUG:boto:Token: None 2024-03-04T16:19:18.057 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj15 2024-03-04T16:19:18.057 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Kx6VC48ss9L6r6EOVP4sWYssApk= 2024-03-04T16:19:18.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Kx6VC48ss9L6r6EOVP4sWYssApk='} 2024-03-04T16:19:18.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe353b9c38e827a0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.066 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.066 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.066 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.066 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.066 DEBUG:boto:Data: 2024-03-04T16:19:18.066 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.066 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.066 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.066 DEBUG:boto:Params: {} 2024-03-04T16:19:18.066 DEBUG:boto:Token: None 2024-03-04T16:19:18.067 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.067 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71f3755e85ad07a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.069 DEBUG:boto:path=/sbnvju-18/obj16 2024-03-04T16:19:18.069 DEBUG:boto:auth_path=/sbnvju-18/obj16 2024-03-04T16:19:18.069 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.069 DEBUG:boto:Path: /sbnvju-18/obj16 2024-03-04T16:19:18.069 DEBUG:boto:Data: 2024-03-04T16:19:18.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.070 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.070 DEBUG:boto:Params: {} 2024-03-04T16:19:18.070 DEBUG:boto:Token: None 2024-03-04T16:19:18.070 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj16 2024-03-04T16:19:18.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Okx0GtoJtWnnUk0r6Xcf0vFoUCc= 2024-03-04T16:19:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Okx0GtoJtWnnUk0r6Xcf0vFoUCc='} 2024-03-04T16:19:18.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094344a59992badee-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.112 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.112 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.112 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.113 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.113 DEBUG:boto:Data: 2024-03-04T16:19:18.113 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.113 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.113 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.113 DEBUG:boto:Params: {} 2024-03-04T16:19:18.113 DEBUG:boto:Token: None 2024-03-04T16:19:18.113 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.114 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000351bc945120736b7-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.116 DEBUG:boto:path=/sbnvju-18/obj17 2024-03-04T16:19:18.116 DEBUG:boto:auth_path=/sbnvju-18/obj17 2024-03-04T16:19:18.116 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.116 DEBUG:boto:Path: /sbnvju-18/obj17 2024-03-04T16:19:18.116 DEBUG:boto:Data: 2024-03-04T16:19:18.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.116 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.116 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.117 DEBUG:boto:Params: {} 2024-03-04T16:19:18.117 DEBUG:boto:Token: None 2024-03-04T16:19:18.117 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj17 2024-03-04T16:19:18.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l7V1qjsiS2WCO5bXaLLV6wAGcVU= 2024-03-04T16:19:18.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l7V1qjsiS2WCO5bXaLLV6wAGcVU='} 2024-03-04T16:19:18.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc50b7b93d5bd679-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.121 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.121 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.121 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.121 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.121 DEBUG:boto:Data: 2024-03-04T16:19:18.121 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.122 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.122 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.122 DEBUG:boto:Params: {} 2024-03-04T16:19:18.122 DEBUG:boto:Token: None 2024-03-04T16:19:18.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd98983824045d9f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.125 DEBUG:boto:path=/sbnvju-18/obj18 2024-03-04T16:19:18.125 DEBUG:boto:auth_path=/sbnvju-18/obj18 2024-03-04T16:19:18.125 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.125 DEBUG:boto:Path: /sbnvju-18/obj18 2024-03-04T16:19:18.125 DEBUG:boto:Data: 2024-03-04T16:19:18.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.125 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.125 DEBUG:boto:Params: {} 2024-03-04T16:19:18.126 DEBUG:boto:Token: None 2024-03-04T16:19:18.126 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj18 2024-03-04T16:19:18.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qAV3phT6mjoGBBojGdu3+0zwkw0= 2024-03-04T16:19:18.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qAV3phT6mjoGBBojGdu3+0zwkw0='} 2024-03-04T16:19:18.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1a3e865d95074f3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.168 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.169 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.169 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.169 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.169 DEBUG:boto:Data: 2024-03-04T16:19:18.169 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.169 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.169 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.169 DEBUG:boto:Params: {} 2024-03-04T16:19:18.169 DEBUG:boto:Token: None 2024-03-04T16:19:18.169 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008deccc513304ab1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.172 DEBUG:boto:path=/sbnvju-18/obj19 2024-03-04T16:19:18.172 DEBUG:boto:auth_path=/sbnvju-18/obj19 2024-03-04T16:19:18.173 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.173 DEBUG:boto:Path: /sbnvju-18/obj19 2024-03-04T16:19:18.173 DEBUG:boto:Data: 2024-03-04T16:19:18.173 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.173 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.173 DEBUG:boto:Params: {} 2024-03-04T16:19:18.173 DEBUG:boto:Token: None 2024-03-04T16:19:18.173 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj19 2024-03-04T16:19:18.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gtc65Zq5Bg7OYOOUDwjBbzmWII0= 2024-03-04T16:19:18.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gtc65Zq5Bg7OYOOUDwjBbzmWII0='} 2024-03-04T16:19:18.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aecf5f30a5cdbb2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.178 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.178 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.178 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.178 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.178 DEBUG:boto:Data: 2024-03-04T16:19:18.178 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.178 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.178 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.178 DEBUG:boto:Params: {} 2024-03-04T16:19:18.178 DEBUG:boto:Token: None 2024-03-04T16:19:18.178 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.179 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e2b1af94744dfcf-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.181 DEBUG:boto:path=/sbnvju-18/obj20 2024-03-04T16:19:18.181 DEBUG:boto:auth_path=/sbnvju-18/obj20 2024-03-04T16:19:18.181 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.181 DEBUG:boto:Path: /sbnvju-18/obj20 2024-03-04T16:19:18.181 DEBUG:boto:Data: 2024-03-04T16:19:18.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.181 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.181 DEBUG:boto:Params: {} 2024-03-04T16:19:18.182 DEBUG:boto:Token: None 2024-03-04T16:19:18.182 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj20 2024-03-04T16:19:18.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wSlQQHgIUTfTqmdnGa0Tw2lLYhA= 2024-03-04T16:19:18.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wSlQQHgIUTfTqmdnGa0Tw2lLYhA='} 2024-03-04T16:19:18.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd10427098df723-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.228 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.228 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.228 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.229 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.229 DEBUG:boto:Data: 2024-03-04T16:19:18.229 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.229 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.229 DEBUG:boto:Params: {} 2024-03-04T16:19:18.229 DEBUG:boto:Token: None 2024-03-04T16:19:18.229 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2b57717e4682760-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.232 DEBUG:boto:path=/sbnvju-18/obj21 2024-03-04T16:19:18.232 DEBUG:boto:auth_path=/sbnvju-18/obj21 2024-03-04T16:19:18.232 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.232 DEBUG:boto:Path: /sbnvju-18/obj21 2024-03-04T16:19:18.232 DEBUG:boto:Data: 2024-03-04T16:19:18.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.232 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.233 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.233 DEBUG:boto:Params: {} 2024-03-04T16:19:18.233 DEBUG:boto:Token: None 2024-03-04T16:19:18.233 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj21 2024-03-04T16:19:18.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5l7qMUoZdFPuF55WoZjpzvTPV9g= 2024-03-04T16:19:18.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5l7qMUoZdFPuF55WoZjpzvTPV9g='} 2024-03-04T16:19:18.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049b6bc2ed11f8325-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.238 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.238 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.238 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.238 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.238 DEBUG:boto:Data: 2024-03-04T16:19:18.238 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.239 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.239 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.239 DEBUG:boto:Params: {} 2024-03-04T16:19:18.239 DEBUG:boto:Token: None 2024-03-04T16:19:18.239 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.239 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bedf155154743fc-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.241 DEBUG:boto:path=/sbnvju-18/obj22 2024-03-04T16:19:18.241 DEBUG:boto:auth_path=/sbnvju-18/obj22 2024-03-04T16:19:18.241 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.241 DEBUG:boto:Path: /sbnvju-18/obj22 2024-03-04T16:19:18.241 DEBUG:boto:Data: 2024-03-04T16:19:18.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.242 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.242 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.242 DEBUG:boto:Params: {} 2024-03-04T16:19:18.242 DEBUG:boto:Token: None 2024-03-04T16:19:18.242 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj22 2024-03-04T16:19:18.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZnT/GJp9zqw8BCqedcBPzvShpgo= 2024-03-04T16:19:18.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZnT/GJp9zqw8BCqedcBPzvShpgo='} 2024-03-04T16:19:18.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b648164d972195-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.284 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.285 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.285 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.285 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.285 DEBUG:boto:Data: 2024-03-04T16:19:18.285 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.285 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.285 DEBUG:boto:Params: {} 2024-03-04T16:19:18.285 DEBUG:boto:Token: None 2024-03-04T16:19:18.285 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcfd0018186a8f3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.288 DEBUG:boto:path=/sbnvju-18/obj23 2024-03-04T16:19:18.288 DEBUG:boto:auth_path=/sbnvju-18/obj23 2024-03-04T16:19:18.288 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.288 DEBUG:boto:Path: /sbnvju-18/obj23 2024-03-04T16:19:18.288 DEBUG:boto:Data: 2024-03-04T16:19:18.288 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.288 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.289 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.289 DEBUG:boto:Params: {} 2024-03-04T16:19:18.289 DEBUG:boto:Token: None 2024-03-04T16:19:18.289 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj23 2024-03-04T16:19:18.289 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IxZW3y0oqQ36fwGqX/brWrHphc0= 2024-03-04T16:19:18.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IxZW3y0oqQ36fwGqX/brWrHphc0='} 2024-03-04T16:19:18.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e67cc168770c2043-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.292 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.293 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.293 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.293 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.293 DEBUG:boto:Data: 2024-03-04T16:19:18.293 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.293 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.293 DEBUG:boto:Params: {} 2024-03-04T16:19:18.293 DEBUG:boto:Token: None 2024-03-04T16:19:18.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.293 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081039a62a30d57d9-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.296 DEBUG:boto:path=/sbnvju-18/obj24 2024-03-04T16:19:18.296 DEBUG:boto:auth_path=/sbnvju-18/obj24 2024-03-04T16:19:18.296 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.296 DEBUG:boto:Path: /sbnvju-18/obj24 2024-03-04T16:19:18.296 DEBUG:boto:Data: 2024-03-04T16:19:18.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.296 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.296 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.296 DEBUG:boto:Params: {} 2024-03-04T16:19:18.297 DEBUG:boto:Token: None 2024-03-04T16:19:18.297 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj24 2024-03-04T16:19:18.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dG9NhdcBSUgG+nBaBLcQxv3paqQ= 2024-03-04T16:19:18.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dG9NhdcBSUgG+nBaBLcQxv3paqQ='} 2024-03-04T16:19:18.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016beb55029e8f3c0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.301 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.301 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.301 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.301 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.301 DEBUG:boto:Data: 2024-03-04T16:19:18.301 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.301 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.301 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.301 DEBUG:boto:Params: {} 2024-03-04T16:19:18.302 DEBUG:boto:Token: None 2024-03-04T16:19:18.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000978aff216ebe4b2b-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.304 DEBUG:boto:path=/sbnvju-18/obj25 2024-03-04T16:19:18.304 DEBUG:boto:auth_path=/sbnvju-18/obj25 2024-03-04T16:19:18.304 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.304 DEBUG:boto:Path: /sbnvju-18/obj25 2024-03-04T16:19:18.304 DEBUG:boto:Data: 2024-03-04T16:19:18.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.305 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.305 DEBUG:boto:Params: {} 2024-03-04T16:19:18.305 DEBUG:boto:Token: None 2024-03-04T16:19:18.305 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj25 2024-03-04T16:19:18.305 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TZLjbfuk3uBw/1cPbP3TDeyYegI= 2024-03-04T16:19:18.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TZLjbfuk3uBw/1cPbP3TDeyYegI='} 2024-03-04T16:19:18.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a96d3dade0269946-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.348 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.349 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.349 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.349 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.349 DEBUG:boto:Data: 2024-03-04T16:19:18.349 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.349 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.349 DEBUG:boto:Params: {} 2024-03-04T16:19:18.349 DEBUG:boto:Token: None 2024-03-04T16:19:18.349 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee356aa831089309-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.352 DEBUG:boto:path=/sbnvju-18/obj26 2024-03-04T16:19:18.352 DEBUG:boto:auth_path=/sbnvju-18/obj26 2024-03-04T16:19:18.352 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.353 DEBUG:boto:Path: /sbnvju-18/obj26 2024-03-04T16:19:18.353 DEBUG:boto:Data: 2024-03-04T16:19:18.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.353 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.353 DEBUG:boto:Params: {} 2024-03-04T16:19:18.353 DEBUG:boto:Token: None 2024-03-04T16:19:18.353 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj26 2024-03-04T16:19:18.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2yFCAsg/b7Qg0lcLJ5XE3BnbeWw= 2024-03-04T16:19:18.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2yFCAsg/b7Qg0lcLJ5XE3BnbeWw='} 2024-03-04T16:19:18.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000223f3cd34268346a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.357 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.357 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.357 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.357 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.357 DEBUG:boto:Data: 2024-03-04T16:19:18.357 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.358 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.358 DEBUG:boto:Params: {} 2024-03-04T16:19:18.358 DEBUG:boto:Token: None 2024-03-04T16:19:18.358 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.358 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000623a73684df5410f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.360 DEBUG:boto:path=/sbnvju-18/obj27 2024-03-04T16:19:18.360 DEBUG:boto:auth_path=/sbnvju-18/obj27 2024-03-04T16:19:18.360 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.360 DEBUG:boto:Path: /sbnvju-18/obj27 2024-03-04T16:19:18.360 DEBUG:boto:Data: 2024-03-04T16:19:18.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.361 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.361 DEBUG:boto:Params: {} 2024-03-04T16:19:18.361 DEBUG:boto:Token: None 2024-03-04T16:19:18.361 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj27 2024-03-04T16:19:18.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XCfoafgARPiS/NjnmEbvtU4mGgM= 2024-03-04T16:19:18.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XCfoafgARPiS/NjnmEbvtU4mGgM='} 2024-03-04T16:19:18.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d6a2c56b5637119-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.365 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.365 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.365 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.365 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.365 DEBUG:boto:Data: 2024-03-04T16:19:18.365 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.365 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.365 DEBUG:boto:Params: {} 2024-03-04T16:19:18.365 DEBUG:boto:Token: None 2024-03-04T16:19:18.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4f4b99cc9bc694f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.368 DEBUG:boto:path=/sbnvju-18/obj28 2024-03-04T16:19:18.368 DEBUG:boto:auth_path=/sbnvju-18/obj28 2024-03-04T16:19:18.368 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.368 DEBUG:boto:Path: /sbnvju-18/obj28 2024-03-04T16:19:18.368 DEBUG:boto:Data: 2024-03-04T16:19:18.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.368 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.368 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.368 DEBUG:boto:Params: {} 2024-03-04T16:19:18.368 DEBUG:boto:Token: None 2024-03-04T16:19:18.369 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj28 2024-03-04T16:19:18.369 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SmVgqW05TzJIPNr9VPDy87CM8II= 2024-03-04T16:19:18.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SmVgqW05TzJIPNr9VPDy87CM8II='} 2024-03-04T16:19:18.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e9cc83a33db7d8-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.412 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.412 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.412 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.413 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.413 DEBUG:boto:Data: 2024-03-04T16:19:18.413 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.413 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.413 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.413 DEBUG:boto:Params: {} 2024-03-04T16:19:18.413 DEBUG:boto:Token: None 2024-03-04T16:19:18.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff53156e51ff9579-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.416 DEBUG:boto:path=/sbnvju-18/obj29 2024-03-04T16:19:18.416 DEBUG:boto:auth_path=/sbnvju-18/obj29 2024-03-04T16:19:18.416 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.416 DEBUG:boto:Path: /sbnvju-18/obj29 2024-03-04T16:19:18.417 DEBUG:boto:Data: 2024-03-04T16:19:18.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.417 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.417 DEBUG:boto:Params: {} 2024-03-04T16:19:18.417 DEBUG:boto:Token: None 2024-03-04T16:19:18.417 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj29 2024-03-04T16:19:18.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EXGo1FF95svt9jegJ1H+LG2vlDw= 2024-03-04T16:19:18.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EXGo1FF95svt9jegJ1H+LG2vlDw='} 2024-03-04T16:19:18.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1a8fea8152cfc1b-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.421 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.421 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.421 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.422 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.422 DEBUG:boto:Data: 2024-03-04T16:19:18.422 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.422 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.422 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.422 DEBUG:boto:Params: {} 2024-03-04T16:19:18.422 DEBUG:boto:Token: None 2024-03-04T16:19:18.422 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5a8f6fd3963352-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.425 DEBUG:boto:path=/sbnvju-18/obj30 2024-03-04T16:19:18.425 DEBUG:boto:auth_path=/sbnvju-18/obj30 2024-03-04T16:19:18.425 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.425 DEBUG:boto:Path: /sbnvju-18/obj30 2024-03-04T16:19:18.425 DEBUG:boto:Data: 2024-03-04T16:19:18.425 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.425 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.425 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.425 DEBUG:boto:Params: {} 2024-03-04T16:19:18.426 DEBUG:boto:Token: None 2024-03-04T16:19:18.426 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj30 2024-03-04T16:19:18.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S//ieUp+uELlCj6kO18UJvdoeB4= 2024-03-04T16:19:18.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S//ieUp+uELlCj6kO18UJvdoeB4='} 2024-03-04T16:19:18.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faf8f0ab6f2127a4-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.468 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.468 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.468 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.469 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.469 DEBUG:boto:Data: 2024-03-04T16:19:18.469 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.469 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.469 DEBUG:boto:Params: {} 2024-03-04T16:19:18.469 DEBUG:boto:Token: None 2024-03-04T16:19:18.469 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.469 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de257ad6e4b4fb80-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.472 DEBUG:boto:path=/sbnvju-18/obj31 2024-03-04T16:19:18.472 DEBUG:boto:auth_path=/sbnvju-18/obj31 2024-03-04T16:19:18.472 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.472 DEBUG:boto:Path: /sbnvju-18/obj31 2024-03-04T16:19:18.472 DEBUG:boto:Data: 2024-03-04T16:19:18.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.472 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.472 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.472 DEBUG:boto:Params: {} 2024-03-04T16:19:18.473 DEBUG:boto:Token: None 2024-03-04T16:19:18.473 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj31 2024-03-04T16:19:18.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EYn/i/lc42GBqcV/KHpzJh3cFXs= 2024-03-04T16:19:18.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EYn/i/lc42GBqcV/KHpzJh3cFXs='} 2024-03-04T16:19:18.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020c7a2d52208b57d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.477 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.477 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.477 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.477 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.477 DEBUG:boto:Data: 2024-03-04T16:19:18.477 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.477 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.477 DEBUG:boto:Params: {} 2024-03-04T16:19:18.477 DEBUG:boto:Token: None 2024-03-04T16:19:18.478 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020a31df6677c2fe1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.480 DEBUG:boto:path=/sbnvju-18/obj32 2024-03-04T16:19:18.480 DEBUG:boto:auth_path=/sbnvju-18/obj32 2024-03-04T16:19:18.480 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.480 DEBUG:boto:Path: /sbnvju-18/obj32 2024-03-04T16:19:18.480 DEBUG:boto:Data: 2024-03-04T16:19:18.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.481 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.481 DEBUG:boto:Params: {} 2024-03-04T16:19:18.481 DEBUG:boto:Token: None 2024-03-04T16:19:18.481 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj32 2024-03-04T16:19:18.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:odNQOb0+dJ7me6ESNsc0XkRa5MM= 2024-03-04T16:19:18.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:odNQOb0+dJ7me6ESNsc0XkRa5MM='} 2024-03-04T16:19:18.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0976a29871aaed2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.485 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.485 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.485 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.485 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.485 DEBUG:boto:Data: 2024-03-04T16:19:18.485 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.485 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.485 DEBUG:boto:Params: {} 2024-03-04T16:19:18.486 DEBUG:boto:Token: None 2024-03-04T16:19:18.486 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c17526cedebc3a1d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.488 DEBUG:boto:path=/sbnvju-18/obj33 2024-03-04T16:19:18.488 DEBUG:boto:auth_path=/sbnvju-18/obj33 2024-03-04T16:19:18.488 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.488 DEBUG:boto:Path: /sbnvju-18/obj33 2024-03-04T16:19:18.488 DEBUG:boto:Data: 2024-03-04T16:19:18.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.488 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.489 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.489 DEBUG:boto:Params: {} 2024-03-04T16:19:18.489 DEBUG:boto:Token: None 2024-03-04T16:19:18.489 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj33 2024-03-04T16:19:18.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:n/fD0jr1lDDBwgWhAWs0gxlXFSw= 2024-03-04T16:19:18.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n/fD0jr1lDDBwgWhAWs0gxlXFSw='} 2024-03-04T16:19:18.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003683ced434f14224-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.532 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.533 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.533 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.533 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.533 DEBUG:boto:Data: 2024-03-04T16:19:18.533 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.533 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.533 DEBUG:boto:Params: {} 2024-03-04T16:19:18.533 DEBUG:boto:Token: None 2024-03-04T16:19:18.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8a04534c1c19f6c-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.536 DEBUG:boto:path=/sbnvju-18/obj34 2024-03-04T16:19:18.536 DEBUG:boto:auth_path=/sbnvju-18/obj34 2024-03-04T16:19:18.536 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.536 DEBUG:boto:Path: /sbnvju-18/obj34 2024-03-04T16:19:18.536 DEBUG:boto:Data: 2024-03-04T16:19:18.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.536 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.537 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.537 DEBUG:boto:Params: {} 2024-03-04T16:19:18.537 DEBUG:boto:Token: None 2024-03-04T16:19:18.537 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj34 2024-03-04T16:19:18.537 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XGU73oSBGV6xuPxrf6uLIqivwd4= 2024-03-04T16:19:18.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XGU73oSBGV6xuPxrf6uLIqivwd4='} 2024-03-04T16:19:18.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe145d285d927df6-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.544 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.544 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.544 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.544 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.544 DEBUG:boto:Data: 2024-03-04T16:19:18.544 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.544 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.544 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.544 DEBUG:boto:Params: {} 2024-03-04T16:19:18.545 DEBUG:boto:Token: None 2024-03-04T16:19:18.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7b965409a6aa37f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.547 DEBUG:boto:path=/sbnvju-18/obj35 2024-03-04T16:19:18.547 DEBUG:boto:auth_path=/sbnvju-18/obj35 2024-03-04T16:19:18.547 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.547 DEBUG:boto:Path: /sbnvju-18/obj35 2024-03-04T16:19:18.547 DEBUG:boto:Data: 2024-03-04T16:19:18.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.548 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.548 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.548 DEBUG:boto:Params: {} 2024-03-04T16:19:18.548 DEBUG:boto:Token: None 2024-03-04T16:19:18.548 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj35 2024-03-04T16:19:18.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:icdSXG0hOyiXPLsufL9CnOioQEQ= 2024-03-04T16:19:18.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:icdSXG0hOyiXPLsufL9CnOioQEQ='} 2024-03-04T16:19:18.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000084afb26c43196c8-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.552 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.552 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.552 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.552 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.553 DEBUG:boto:Data: 2024-03-04T16:19:18.553 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.553 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.553 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.553 DEBUG:boto:Params: {} 2024-03-04T16:19:18.553 DEBUG:boto:Token: None 2024-03-04T16:19:18.553 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034c5645ac36a4cbb-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.555 DEBUG:boto:path=/sbnvju-18/obj36 2024-03-04T16:19:18.555 DEBUG:boto:auth_path=/sbnvju-18/obj36 2024-03-04T16:19:18.556 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.556 DEBUG:boto:Path: /sbnvju-18/obj36 2024-03-04T16:19:18.556 DEBUG:boto:Data: 2024-03-04T16:19:18.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.556 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.556 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.556 DEBUG:boto:Params: {} 2024-03-04T16:19:18.556 DEBUG:boto:Token: None 2024-03-04T16:19:18.556 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj36 2024-03-04T16:19:18.557 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvG9tZ06VqXPe/Xbh+QLh++RAfQ= 2024-03-04T16:19:18.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvG9tZ06VqXPe/Xbh+QLh++RAfQ='} 2024-03-04T16:19:18.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c0d19e4f00a5bad-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.601 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.601 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.601 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.601 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.601 DEBUG:boto:Data: 2024-03-04T16:19:18.601 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.601 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.602 DEBUG:boto:Params: {} 2024-03-04T16:19:18.602 DEBUG:boto:Token: None 2024-03-04T16:19:18.602 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d3d6125fca73a2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.604 DEBUG:boto:path=/sbnvju-18/obj37 2024-03-04T16:19:18.604 DEBUG:boto:auth_path=/sbnvju-18/obj37 2024-03-04T16:19:18.604 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.605 DEBUG:boto:Path: /sbnvju-18/obj37 2024-03-04T16:19:18.605 DEBUG:boto:Data: 2024-03-04T16:19:18.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.605 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.605 DEBUG:boto:Params: {} 2024-03-04T16:19:18.605 DEBUG:boto:Token: None 2024-03-04T16:19:18.605 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj37 2024-03-04T16:19:18.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uWrkCOOCYBEyTlITtLrnkcdBd3k= 2024-03-04T16:19:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uWrkCOOCYBEyTlITtLrnkcdBd3k='} 2024-03-04T16:19:18.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2c0de123d661b55-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.609 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.609 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.609 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.609 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.609 DEBUG:boto:Data: 2024-03-04T16:19:18.609 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.610 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.610 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.610 DEBUG:boto:Params: {} 2024-03-04T16:19:18.610 DEBUG:boto:Token: None 2024-03-04T16:19:18.610 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.610 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af36e40dbc939550-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.613 DEBUG:boto:path=/sbnvju-18/obj38 2024-03-04T16:19:18.613 DEBUG:boto:auth_path=/sbnvju-18/obj38 2024-03-04T16:19:18.613 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.613 DEBUG:boto:Path: /sbnvju-18/obj38 2024-03-04T16:19:18.613 DEBUG:boto:Data: 2024-03-04T16:19:18.613 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.613 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.613 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.613 DEBUG:boto:Params: {} 2024-03-04T16:19:18.614 DEBUG:boto:Token: None 2024-03-04T16:19:18.614 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj38 2024-03-04T16:19:18.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7dFUP8mYtrih5BY+kTAWFRSTPU0= 2024-03-04T16:19:18.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7dFUP8mYtrih5BY+kTAWFRSTPU0='} 2024-03-04T16:19:18.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043b5a0b183bc93db-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.656 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.656 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.656 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.657 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.657 DEBUG:boto:Data: 2024-03-04T16:19:18.657 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.657 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.657 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.657 DEBUG:boto:Params: {} 2024-03-04T16:19:18.657 DEBUG:boto:Token: None 2024-03-04T16:19:18.657 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.657 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001610e76087499bad-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.660 DEBUG:boto:path=/sbnvju-18/obj39 2024-03-04T16:19:18.660 DEBUG:boto:auth_path=/sbnvju-18/obj39 2024-03-04T16:19:18.660 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.660 DEBUG:boto:Path: /sbnvju-18/obj39 2024-03-04T16:19:18.660 DEBUG:boto:Data: 2024-03-04T16:19:18.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.660 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.660 DEBUG:boto:Params: {} 2024-03-04T16:19:18.660 DEBUG:boto:Token: None 2024-03-04T16:19:18.661 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj39 2024-03-04T16:19:18.661 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rnETUFYZlMvzDpu8hoyRnz7FE1k= 2024-03-04T16:19:18.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rnETUFYZlMvzDpu8hoyRnz7FE1k='} 2024-03-04T16:19:18.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058ff2d274ddfa5cd-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.664 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.665 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.665 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.665 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.665 DEBUG:boto:Data: 2024-03-04T16:19:18.665 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.665 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.665 DEBUG:boto:Params: {} 2024-03-04T16:19:18.665 DEBUG:boto:Token: None 2024-03-04T16:19:18.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.665 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8c50f6cffdeab58-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.668 DEBUG:boto:path=/sbnvju-18/obj40 2024-03-04T16:19:18.668 DEBUG:boto:auth_path=/sbnvju-18/obj40 2024-03-04T16:19:18.668 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.668 DEBUG:boto:Path: /sbnvju-18/obj40 2024-03-04T16:19:18.668 DEBUG:boto:Data: 2024-03-04T16:19:18.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.668 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.668 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.668 DEBUG:boto:Params: {} 2024-03-04T16:19:18.669 DEBUG:boto:Token: None 2024-03-04T16:19:18.669 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj40 2024-03-04T16:19:18.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QQ2NL8PZYUyO9avhLTdJ4TXoQg= 2024-03-04T16:19:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QQ2NL8PZYUyO9avhLTdJ4TXoQg='} 2024-03-04T16:19:18.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000befbf20221ae0d50-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.674 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.674 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.674 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.674 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.675 DEBUG:boto:Data: 2024-03-04T16:19:18.675 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.675 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.675 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.675 DEBUG:boto:Params: {} 2024-03-04T16:19:18.675 DEBUG:boto:Token: None 2024-03-04T16:19:18.675 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.675 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092dd1c1119493219-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.678 DEBUG:boto:path=/sbnvju-18/obj41 2024-03-04T16:19:18.678 DEBUG:boto:auth_path=/sbnvju-18/obj41 2024-03-04T16:19:18.678 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.678 DEBUG:boto:Path: /sbnvju-18/obj41 2024-03-04T16:19:18.678 DEBUG:boto:Data: 2024-03-04T16:19:18.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.678 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.678 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.678 DEBUG:boto:Params: {} 2024-03-04T16:19:18.678 DEBUG:boto:Token: None 2024-03-04T16:19:18.679 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj41 2024-03-04T16:19:18.679 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UeBQ140qJnkp+9awYntKh7F91Jc= 2024-03-04T16:19:18.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UeBQ140qJnkp+9awYntKh7F91Jc='} 2024-03-04T16:19:18.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec929a21674376fb-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.724 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.724 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.725 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.725 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.725 DEBUG:boto:Data: 2024-03-04T16:19:18.725 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.725 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.725 DEBUG:boto:Params: {} 2024-03-04T16:19:18.725 DEBUG:boto:Token: None 2024-03-04T16:19:18.725 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003283f65521336aae-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.728 DEBUG:boto:path=/sbnvju-18/obj42 2024-03-04T16:19:18.728 DEBUG:boto:auth_path=/sbnvju-18/obj42 2024-03-04T16:19:18.728 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.728 DEBUG:boto:Path: /sbnvju-18/obj42 2024-03-04T16:19:18.728 DEBUG:boto:Data: 2024-03-04T16:19:18.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.728 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.728 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.729 DEBUG:boto:Params: {} 2024-03-04T16:19:18.729 DEBUG:boto:Token: None 2024-03-04T16:19:18.729 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj42 2024-03-04T16:19:18.729 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tlIsfY17t+pl7bLnH35/WxDTyXA= 2024-03-04T16:19:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tlIsfY17t+pl7bLnH35/WxDTyXA='} 2024-03-04T16:19:18.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094057f10999e8fa4-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.733 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.733 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.733 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.733 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.733 DEBUG:boto:Data: 2024-03-04T16:19:18.733 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.733 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.734 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.734 DEBUG:boto:Params: {} 2024-03-04T16:19:18.734 DEBUG:boto:Token: None 2024-03-04T16:19:18.734 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.734 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a6845cb7dddb93f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.736 DEBUG:boto:path=/sbnvju-18/obj43 2024-03-04T16:19:18.737 DEBUG:boto:auth_path=/sbnvju-18/obj43 2024-03-04T16:19:18.737 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.737 DEBUG:boto:Path: /sbnvju-18/obj43 2024-03-04T16:19:18.737 DEBUG:boto:Data: 2024-03-04T16:19:18.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.737 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.737 DEBUG:boto:Params: {} 2024-03-04T16:19:18.737 DEBUG:boto:Token: None 2024-03-04T16:19:18.738 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj43 2024-03-04T16:19:18.738 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PzZHP4Shxg/s/D4GyZgFwW9l/Dg= 2024-03-04T16:19:18.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PzZHP4Shxg/s/D4GyZgFwW9l/Dg='} 2024-03-04T16:19:18.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a72fc56cc5d84f8a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.781 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.781 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.781 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.781 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.782 DEBUG:boto:Data: 2024-03-04T16:19:18.782 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.782 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.782 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.782 DEBUG:boto:Params: {} 2024-03-04T16:19:18.783 DEBUG:boto:Token: None 2024-03-04T16:19:18.783 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.783 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6fd05d33f22670f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.786 DEBUG:boto:path=/sbnvju-18/obj44 2024-03-04T16:19:18.786 DEBUG:boto:auth_path=/sbnvju-18/obj44 2024-03-04T16:19:18.787 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.787 DEBUG:boto:Path: /sbnvju-18/obj44 2024-03-04T16:19:18.787 DEBUG:boto:Data: 2024-03-04T16:19:18.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.787 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.787 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.788 DEBUG:boto:Params: {} 2024-03-04T16:19:18.788 DEBUG:boto:Token: None 2024-03-04T16:19:18.788 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj44 2024-03-04T16:19:18.788 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JIdll/AQBKmprpTpHe7Ja4npEIg= 2024-03-04T16:19:18.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JIdll/AQBKmprpTpHe7Ja4npEIg='} 2024-03-04T16:19:18.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e455b697dd3a0aa3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.794 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.794 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.794 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.795 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.795 DEBUG:boto:Data: 2024-03-04T16:19:18.795 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.795 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.795 DEBUG:boto:Params: {} 2024-03-04T16:19:18.795 DEBUG:boto:Token: None 2024-03-04T16:19:18.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2bb27d72f7ff1ca-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.798 DEBUG:boto:path=/sbnvju-18/obj45 2024-03-04T16:19:18.798 DEBUG:boto:auth_path=/sbnvju-18/obj45 2024-03-04T16:19:18.798 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.798 DEBUG:boto:Path: /sbnvju-18/obj45 2024-03-04T16:19:18.798 DEBUG:boto:Data: 2024-03-04T16:19:18.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.798 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.798 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.799 DEBUG:boto:Params: {} 2024-03-04T16:19:18.799 DEBUG:boto:Token: None 2024-03-04T16:19:18.799 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj45 2024-03-04T16:19:18.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CjGGcwyY4nkuK6xPWSGz7DMqu+o= 2024-03-04T16:19:18.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CjGGcwyY4nkuK6xPWSGz7DMqu+o='} 2024-03-04T16:19:18.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047c7df1586efccf0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.804 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.804 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.804 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.804 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.804 DEBUG:boto:Data: 2024-03-04T16:19:18.804 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.804 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.804 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.804 DEBUG:boto:Params: {} 2024-03-04T16:19:18.805 DEBUG:boto:Token: None 2024-03-04T16:19:18.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018eefd01008d0083-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.807 DEBUG:boto:path=/sbnvju-18/obj46 2024-03-04T16:19:18.807 DEBUG:boto:auth_path=/sbnvju-18/obj46 2024-03-04T16:19:18.807 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.808 DEBUG:boto:Path: /sbnvju-18/obj46 2024-03-04T16:19:18.808 DEBUG:boto:Data: 2024-03-04T16:19:18.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.808 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.808 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.808 DEBUG:boto:Params: {} 2024-03-04T16:19:18.808 DEBUG:boto:Token: None 2024-03-04T16:19:18.808 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj46 2024-03-04T16:19:18.808 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RF5bvjuzS7BXJPfE/BEAXIhjR3w= 2024-03-04T16:19:18.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RF5bvjuzS7BXJPfE/BEAXIhjR3w='} 2024-03-04T16:19:18.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a07068af66128157-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.812 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.812 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.812 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.812 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.813 DEBUG:boto:Data: 2024-03-04T16:19:18.813 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.813 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.813 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.813 DEBUG:boto:Params: {} 2024-03-04T16:19:18.813 DEBUG:boto:Token: None 2024-03-04T16:19:18.813 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.813 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c530f1990bcb829-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.815 DEBUG:boto:path=/sbnvju-18/obj47 2024-03-04T16:19:18.815 DEBUG:boto:auth_path=/sbnvju-18/obj47 2024-03-04T16:19:18.816 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.816 DEBUG:boto:Path: /sbnvju-18/obj47 2024-03-04T16:19:18.816 DEBUG:boto:Data: 2024-03-04T16:19:18.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.816 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.816 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.816 DEBUG:boto:Params: {} 2024-03-04T16:19:18.816 DEBUG:boto:Token: None 2024-03-04T16:19:18.816 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj47 2024-03-04T16:19:18.816 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vKA6etbYid0oWSeUqLfJvgRHSlg= 2024-03-04T16:19:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vKA6etbYid0oWSeUqLfJvgRHSlg='} 2024-03-04T16:19:18.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000371a95ac8847452e-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.863 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.863 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.863 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.863 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.863 DEBUG:boto:Data: 2024-03-04T16:19:18.864 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.864 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.864 DEBUG:boto:Params: {} 2024-03-04T16:19:18.864 DEBUG:boto:Token: None 2024-03-04T16:19:18.864 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.864 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000394d9d18c7d3f24d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.866 DEBUG:boto:path=/sbnvju-18/obj48 2024-03-04T16:19:18.867 DEBUG:boto:auth_path=/sbnvju-18/obj48 2024-03-04T16:19:18.867 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.867 DEBUG:boto:Path: /sbnvju-18/obj48 2024-03-04T16:19:18.867 DEBUG:boto:Data: 2024-03-04T16:19:18.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.867 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.867 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.867 DEBUG:boto:Params: {} 2024-03-04T16:19:18.867 DEBUG:boto:Token: None 2024-03-04T16:19:18.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj48 2024-03-04T16:19:18.868 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZAoYtZQ8mL8mIvSiHPUse1f+nCg= 2024-03-04T16:19:18.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZAoYtZQ8mL8mIvSiHPUse1f+nCg='} 2024-03-04T16:19:18.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003797e2294aa358d1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.871 DEBUG:boto:path=/sbnvju-18/ 2024-03-04T16:19:18.872 DEBUG:boto:auth_path=/sbnvju-18/ 2024-03-04T16:19:18.872 DEBUG:boto:Method: HEAD 2024-03-04T16:19:18.872 DEBUG:boto:Path: /sbnvju-18/ 2024-03-04T16:19:18.872 DEBUG:boto:Data: 2024-03-04T16:19:18.872 DEBUG:boto:Headers: {} 2024-03-04T16:19:18.872 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.872 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.872 DEBUG:boto:Params: {} 2024-03-04T16:19:18.872 DEBUG:boto:Token: None 2024-03-04T16:19:18.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/ 2024-03-04T16:19:18.873 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T16:19:18.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T16:19:18.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb87cef1942b2a09-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.875 DEBUG:boto:path=/sbnvju-18/obj49 2024-03-04T16:19:18.875 DEBUG:boto:auth_path=/sbnvju-18/obj49 2024-03-04T16:19:18.875 DEBUG:boto:Method: PUT 2024-03-04T16:19:18.875 DEBUG:boto:Path: /sbnvju-18/obj49 2024-03-04T16:19:18.875 DEBUG:boto:Data: 2024-03-04T16:19:18.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T16:19:18.875 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:19:18.875 DEBUG:boto:Port: 8001 2024-03-04T16:19:18.875 DEBUG:boto:Params: {} 2024-03-04T16:19:18.875 DEBUG:boto:Token: None 2024-03-04T16:19:18.876 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 16:19:18 GMT /sbnvju-18/obj49 2024-03-04T16:19:18.876 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z5k1E4aVHjTYke7olWmOfBV/CLM= 2024-03-04T16:19:18.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z5k1E4aVHjTYke7olWmOfBV/CLM='} 2024-03-04T16:19:18.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee75e57fc2885c16-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:19:18.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:18.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:18.880 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:19:18.926 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:19:18.926 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:19:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:19:18.990 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709569155.099706_86.1","last_update":"2024-03-04T16:19:15.099706Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:19:18.991 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:19:18.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:19:18.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:18.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:18.991 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:19:19.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:19:19.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:19:19.098 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:19.098+0000 7f5e5711f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:19:19.099 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:19.099+0000 7f5e5711f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:19:19.118 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T16:19:19.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T16:19:19.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569155.099706_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:19:15.099706Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:19:19.119 DEBUG:rgw_multi.tests:log_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:19:19.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:19:19.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:19:19.119 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:19:19.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:19.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:19:19.120 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:19:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:19:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:19:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:19.214+0000 7f6cd2813800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:19:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:19:19.215+0000 7f6cd2813800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:19:19.234 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T16:19:19.234 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T16:19:19.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569155.099706_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:19:15.099706Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:19:19.234 DEBUG:rgw_multi.tests:log_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:19:19.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:19:19.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:19:19.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:19:19.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:19:19.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:19:19.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:19:19.290 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:19:19.337 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:19.337+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:24.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:24.338+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:29.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:29.338+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:34.339+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:39.340+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:44.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:44.340+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:49.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:49.341+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:54.342 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:54.342+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:19:59.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:19:59.343+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:04.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:04.344+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:09.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:09.344+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:14.345+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:19.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:19.345+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:24.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:24.346+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:29.347+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:34.348+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:39.348+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:44.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:44.349+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:49.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:49.350+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:54.356 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:54.351+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:20:59.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:20:59.352+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:04.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:04.351+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:09.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:09.352+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:14.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:14.353+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:19.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:19.353+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:24.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:24.354+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:29.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:29.355+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:34.356 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:34.356+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:39.356+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:44.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:44.357+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:49.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:49.358+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:54.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:54.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:21:59.359 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:21:59.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:04.359 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:04.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:09.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:09.360+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:14.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:14.360+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:19.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:19.361+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:24.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:24.362+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:29.362+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:34.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:34.363+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:39.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:39.364+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:44.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:44.364+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:49.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:49.365+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:54.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:54.365+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:22:59.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:22:59.366+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:04.367 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:04.367+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:09.368 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:09.368+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:14.368 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:14.368+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:19.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:19.369+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:24.370 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:24.369+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:29.370+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:34.371 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:34.371+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:39.371+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:44.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:44.373+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:49.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:49.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:54.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:54.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:23:59.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:23:59.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:24:04.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:04.375+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:24:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:09.375+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T16:24:14.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:14.375+0000 7fe4d6e16800 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-03-04T16:24:14.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:14.375+0000 7fe4d6e16800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T16:24:14.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:14.375+0000 7fe4d6e16800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T16:24:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-03-04T16:24:14.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-03-04T16:24:14.423 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-19 2024-03-04T16:24:14.423 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:24:14.423 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:24:14.424 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.424 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:24:14.424 DEBUG:boto:Data: 2024-03-04T16:24:14.424 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.424 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:24:14.425 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.425 DEBUG:boto:Params: {} 2024-03-04T16:24:14.425 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:24:14.425 DEBUG:boto:Token: None 2024-03-04T16:24:14.426 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-19/ 2024-03-04T16:24:14.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kJKtAvN03DcUrVd/PdC61YDTQrE= 2024-03-04T16:24:14.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kJKtAvN03DcUrVd/PdC61YDTQrE=', 'Content-Length': '0'} 2024-03-04T16:24:14.448 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b413e6123c7ebec0-0065e5f5ae-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.448 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-20 2024-03-04T16:24:14.448 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:24:14.448 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:24:14.448 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.448 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:24:14.449 DEBUG:boto:Data: 2024-03-04T16:24:14.449 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:24:14.449 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.449 DEBUG:boto:Params: {} 2024-03-04T16:24:14.449 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:24:14.449 DEBUG:boto:Token: None 2024-03-04T16:24:14.449 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-20/ 2024-03-04T16:24:14.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7Uqs+n7aAsAGSjQBcvhlgdZ5Wzk= 2024-03-04T16:24:14.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7Uqs+n7aAsAGSjQBcvhlgdZ5Wzk=', 'Content-Length': '0'} 2024-03-04T16:24:14.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5b084ff79188b9f-0065e5f5ae-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.479 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-21 2024-03-04T16:24:14.479 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:24:14.479 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:24:14.480 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.480 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:24:14.480 DEBUG:boto:Data: 2024-03-04T16:24:14.480 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:24:14.480 DEBUG:boto:Port: 8001 2024-03-04T16:24:14.480 DEBUG:boto:Params: {} 2024-03-04T16:24:14.480 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:24:14.481 DEBUG:boto:Token: None 2024-03-04T16:24:14.481 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-21/ 2024-03-04T16:24:14.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gFlhJceNjmh8s3HtY/4yy+2RRKQ= 2024-03-04T16:24:14.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gFlhJceNjmh8s3HtY/4yy+2RRKQ=', 'Content-Length': '0'} 2024-03-04T16:24:14.508 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000960a290391ef0af4-0065e5f5ae-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.508 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:24:14.508 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:24:14.509 DEBUG:boto:path=/sbnvju-19/?versioning 2024-03-04T16:24:14.509 DEBUG:boto:auth_path=/sbnvju-19/?versioning 2024-03-04T16:24:14.509 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.509 DEBUG:boto:Path: /sbnvju-19/?versioning 2024-03-04T16:24:14.509 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:24:14.509 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.509 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:24:14.509 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.509 DEBUG:boto:Params: {} 2024-03-04T16:24:14.509 DEBUG:boto:Token: None 2024-03-04T16:24:14.510 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-19/?versioning 2024-03-04T16:24:14.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:74CKvwAE3cVkwQipz+DbJtw0bug= 2024-03-04T16:24:14.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:74CKvwAE3cVkwQipz+DbJtw0bug=', 'Content-Length': '228'} 2024-03-04T16:24:14.515 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aaa3ce5b25f4f386-0065e5f5ae-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.515 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:24:14.515 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:24:14.516 DEBUG:boto:path=/sbnvju-20/?versioning 2024-03-04T16:24:14.516 DEBUG:boto:auth_path=/sbnvju-20/?versioning 2024-03-04T16:24:14.516 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.516 DEBUG:boto:Path: /sbnvju-20/?versioning 2024-03-04T16:24:14.516 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:24:14.516 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:24:14.516 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.516 DEBUG:boto:Params: {} 2024-03-04T16:24:14.516 DEBUG:boto:Token: None 2024-03-04T16:24:14.517 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-20/?versioning 2024-03-04T16:24:14.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pryKeeTrpuKNIlYcHzaIXNt7S/Y= 2024-03-04T16:24:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pryKeeTrpuKNIlYcHzaIXNt7S/Y=', 'Content-Length': '228'} 2024-03-04T16:24:14.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f6a4f86036ccfc86-0065e5f5ae-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.526 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:24:14.526 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:24:14.526 DEBUG:boto:path=/sbnvju-21/?versioning 2024-03-04T16:24:14.526 DEBUG:boto:auth_path=/sbnvju-21/?versioning 2024-03-04T16:24:14.526 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.526 DEBUG:boto:Path: /sbnvju-21/?versioning 2024-03-04T16:24:14.526 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:24:14.526 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:24:14.527 DEBUG:boto:Port: 8001 2024-03-04T16:24:14.527 DEBUG:boto:Params: {} 2024-03-04T16:24:14.527 DEBUG:boto:Token: None 2024-03-04T16:24:14.527 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-21/?versioning 2024-03-04T16:24:14.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sGVtJwCbL++SsiPgCpcwczWpRIY= 2024-03-04T16:24:14.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sGVtJwCbL++SsiPgCpcwczWpRIY=', 'Content-Length': '228'} 2024-03-04T16:24:14.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a273e9bdc6203d19-0065e5f5ae-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.536 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:24:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:14.653 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:24:14.679 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709569454.533720_158.1","last_update":"2024-03-04T16:24:14.533720Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:24:14.680 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709569454.533720_158.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:24:14.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:24:14.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.681 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:24:14.730 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:14.731 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:14.776 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:14.776+0000 7f7d9251f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:24:14.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:14.777+0000 7f7d9251f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:24:14.795 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569454.533720_158.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:24:14.533720Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:24:14Z"} 2024-03-04T16:24:14.796 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569454.533720_158.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:24:14.533720Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:24:14Z"} 2024-03-04T16:24:14.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569454.533720_158.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:24:14.533720Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:24:14.796 DEBUG:rgw_multi.tests:log_status={0: '1_1709569454.533720_158.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:24:14.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569454.533720_158.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:24:14.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:24:14.796 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:24:14.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:24:14.797 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:24:14.844 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:14.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:14.888+0000 7f4cfae14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:24:14.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:14.889+0000 7f4cfae14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:24:14.908 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569454.533720_158.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:24:14.533720Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:24:14Z"} 2024-03-04T16:24:14.908 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569454.533720_158.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:24:14.533720Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:24:14Z"} 2024-03-04T16:24:14.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569454.533720_158.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:24:14.533720Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:24:14.908 DEBUG:rgw_multi.tests:log_status={0: '1_1709569454.533720_158.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:24:14.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569454.533720_158.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:24:14.909 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:24:14.909 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:24:14.909 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:24:14.909 DEBUG:boto:Method: HEAD 2024-03-04T16:24:14.909 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:24:14.909 DEBUG:boto:Data: 2024-03-04T16:24:14.909 DEBUG:boto:Headers: {} 2024-03-04T16:24:14.909 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:24:14.909 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.910 DEBUG:boto:Params: {} 2024-03-04T16:24:14.910 DEBUG:boto:Token: None 2024-03-04T16:24:14.910 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-19/ 2024-03-04T16:24:14.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T0CESb8zX3/yxu44XrL0O10EOx0= 2024-03-04T16:24:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T0CESb8zX3/yxu44XrL0O10EOx0=', 'Content-Length': '0'} 2024-03-04T16:24:14.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd957bdacba5bb6-0065e5f5ae-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.913 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:24:14.913 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:24:14.913 DEBUG:boto:Method: PUT 2024-03-04T16:24:14.913 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:24:14.913 DEBUG:boto:Data: 2024-03-04T16:24:14.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:24:14.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:24:14.913 DEBUG:boto:Port: 8000 2024-03-04T16:24:14.914 DEBUG:boto:Params: {} 2024-03-04T16:24:14.914 DEBUG:boto:Token: None 2024-03-04T16:24:14.914 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:24:14 GMT /sbnvju-19/dummy 2024-03-04T16:24:14.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugVAucYUUWVceyUFAoehHnpiuQ0= 2024-03-04T16:24:14.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:24:14 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugVAucYUUWVceyUFAoehHnpiuQ0='} 2024-03-04T16:24:14.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-amz-request-id', 'tx000006e008c2193df5f42-0065e5f5ae-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:14.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:14.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:14.925 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:24:14.978 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:14.978 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:15.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:15.026+0000 7f553b244800 1 waiting to reach incremental sync.. 2024-03-04T16:24:20.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:20.027+0000 7f553b244800 1 waiting to reach incremental sync.. 2024-03-04T16:24:25.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:25.027+0000 7f553b244800 1 waiting to reach incremental sync.. 2024-03-04T16:24:30.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:30.027+0000 7f553b244800 1 waiting to reach incremental sync.. 2024-03-04T16:24:35.030 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:35.029+0000 7f553b244800 1 bucket sync caught up with source: 2024-03-04T16:24:35.030 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.49.10, , , , , ] 2024-03-04T16:24:35.030 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.47.11, , , , , ] 2024-03-04T16:24:35.030 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:24:35.029+0000 7f553b244800 0 bucket checkpoint complete 2024-03-04T16:24:35.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:35.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:35.053 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:24:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:35.170+0000 7f1b9fc8c800 1 waiting for incremental sync to catch up: 2024-03-04T16:24:35.171 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.47.11, , , , , ] 2024-03-04T16:24:35.171 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.49.10, , , , , ] 2024-03-04T16:24:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:40.172+0000 7f1b9fc8c800 1 waiting for incremental sync to catch up: 2024-03-04T16:24:40.172 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.47.11, , , , , ] 2024-03-04T16:24:40.172 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.49.10, , , , , ] 2024-03-04T16:24:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:45.173+0000 7f1b9fc8c800 1 bucket sync caught up with source: 2024-03-04T16:24:45.174 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.49.10, , , , , ] 2024-03-04T16:24:45.174 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.49.10, , , , , ] 2024-03-04T16:24:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:45.173+0000 7f1b9fc8c800 0 bucket checkpoint complete 2024-03-04T16:24:45.197 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:24:45.197 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:24:45.197 DEBUG:boto:Method: HEAD 2024-03-04T16:24:45.197 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:24:45.197 DEBUG:boto:Data: 2024-03-04T16:24:45.197 DEBUG:boto:Headers: {} 2024-03-04T16:24:45.197 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:24:45.198 DEBUG:boto:Port: 8000 2024-03-04T16:24:45.198 DEBUG:boto:Params: {} 2024-03-04T16:24:45.198 DEBUG:boto:Token: None 2024-03-04T16:24:45.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:24:45 GMT /sbnvju-20/ 2024-03-04T16:24:45.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DWduyVeW2HL7lJhxcxLvR8fThjY= 2024-03-04T16:24:45.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:24:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DWduyVeW2HL7lJhxcxLvR8fThjY=', 'Content-Length': '0'} 2024-03-04T16:24:45.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a50c78621737a8d0-0065e5f5cd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:45.202 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:24:45.202 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:24:45.202 DEBUG:boto:Method: PUT 2024-03-04T16:24:45.202 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:24:45.202 DEBUG:boto:Data: 2024-03-04T16:24:45.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:24:45.202 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:24:45.203 DEBUG:boto:Port: 8000 2024-03-04T16:24:45.203 DEBUG:boto:Params: {} 2024-03-04T16:24:45.203 DEBUG:boto:Token: None 2024-03-04T16:24:45.203 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:24:45 GMT /sbnvju-20/dummy 2024-03-04T16:24:45.203 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nj8Z2mSAZtGFqhaqKoC9sbmmFTQ= 2024-03-04T16:24:45.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:24:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nj8Z2mSAZtGFqhaqKoC9sbmmFTQ='} 2024-03-04T16:24:45.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-amz-request-id', 'tx00000f61afb33f25c079b-0065e5f5cd-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:24:45.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:45.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:45.217 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:24:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:45.332+0000 7f1d9a324800 1 waiting to reach incremental sync.. 2024-03-04T16:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:50.332+0000 7f1d9a324800 1 waiting to reach incremental sync.. 2024-03-04T16:24:55.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:55.334+0000 7f1d9a324800 1 bucket sync caught up with source: 2024-03-04T16:24:55.335 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.39.11, , , , , ] 2024-03-04T16:24:55.335 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000002.39.11, , , , , ] 2024-03-04T16:24:55.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:55.334+0000 7f1d9a324800 0 bucket checkpoint complete 2024-03-04T16:24:55.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:55.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:24:55.357 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:24:55.413 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:24:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:24:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:24:55.464+0000 7f964333c800 1 waiting to reach incremental sync.. 2024-03-04T16:25:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:00.464+0000 7f964333c800 1 waiting to reach incremental sync.. 2024-03-04T16:25:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:05.466+0000 7f964333c800 1 bucket sync caught up with source: 2024-03-04T16:25:05.467 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.39.11, , , , , ] 2024-03-04T16:25:05.467 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000002.39.11, , , , , ] 2024-03-04T16:25:05.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:05.466+0000 7f964333c800 0 bucket checkpoint complete 2024-03-04T16:25:05.491 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:25:05.491 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:25:05.491 DEBUG:boto:Method: HEAD 2024-03-04T16:25:05.491 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:25:05.491 DEBUG:boto:Data: 2024-03-04T16:25:05.491 DEBUG:boto:Headers: {} 2024-03-04T16:25:05.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:05.491 DEBUG:boto:Port: 8001 2024-03-04T16:25:05.492 DEBUG:boto:Params: {} 2024-03-04T16:25:05.492 DEBUG:boto:Token: None 2024-03-04T16:25:05.492 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:05 GMT /sbnvju-21/ 2024-03-04T16:25:05.492 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fi9lvL50EOVQTi+LwC1F8FoSjU0= 2024-03-04T16:25:05.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fi9lvL50EOVQTi+LwC1F8FoSjU0=', 'Content-Length': '0'} 2024-03-04T16:25:05.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091daf2f5b3a03c9d-0065e5f5e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:05.496 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:25:05.496 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:25:05.496 DEBUG:boto:Method: PUT 2024-03-04T16:25:05.496 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:25:05.496 DEBUG:boto:Data: 2024-03-04T16:25:05.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:25:05.496 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:05.496 DEBUG:boto:Port: 8001 2024-03-04T16:25:05.496 DEBUG:boto:Params: {} 2024-03-04T16:25:05.497 DEBUG:boto:Token: None 2024-03-04T16:25:05.497 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:25:05 GMT /sbnvju-21/dummy 2024-03-04T16:25:05.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B1MhUXaONlFzGIjufXdjY6poiuI= 2024-03-04T16:25:05.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B1MhUXaONlFzGIjufXdjY6poiuI='} 2024-03-04T16:25:05.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-amz-request-id', 'tx00000646a7b6983f0583d-0065e5f5e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:05.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:05.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:05.510 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:05.568 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:05.568 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:05.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:05.620+0000 7fa517b33800 1 waiting to reach incremental sync.. 2024-03-04T16:25:10.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:10.620+0000 7fa517b33800 1 waiting to reach incremental sync.. 2024-03-04T16:25:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:15.622+0000 7fa517b33800 1 bucket sync caught up with source: 2024-03-04T16:25:15.623 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.19.11, , , , , ] 2024-03-04T16:25:15.623 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000002.19.11, , , , , ] 2024-03-04T16:25:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:15.622+0000 7fa517b33800 0 bucket checkpoint complete 2024-03-04T16:25:15.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:15.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:15.647 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:15.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:15.762+0000 7f113472f800 1 waiting to reach incremental sync.. 2024-03-04T16:25:20.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:20.762+0000 7f113472f800 1 waiting to reach incremental sync.. 2024-03-04T16:25:25.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:25.763+0000 7f113472f800 1 waiting to reach incremental sync.. 2024-03-04T16:25:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:30.764+0000 7f113472f800 1 waiting to reach incremental sync.. 2024-03-04T16:25:35.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:35.765+0000 7f113472f800 1 bucket sync caught up with source: 2024-03-04T16:25:35.766 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.21.10, , , , , ] 2024-03-04T16:25:35.766 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.19.11, , , , , ] 2024-03-04T16:25:35.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:35.765+0000 7f113472f800 0 bucket checkpoint complete 2024-03-04T16:25:35.790 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:25:35.790 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:25:35.791 DEBUG:boto:Method: HEAD 2024-03-04T16:25:35.791 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:25:35.791 DEBUG:boto:Data: 2024-03-04T16:25:35.791 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.791 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.791 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.791 DEBUG:boto:Params: {} 2024-03-04T16:25:35.791 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:25:35.792 DEBUG:boto:Token: None 2024-03-04T16:25:35.792 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/ 2024-03-04T16:25:35.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU= 2024-03-04T16:25:35.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU=', 'Content-Length': '0'} 2024-03-04T16:25:35.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1649a5ae2144484-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.797 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.798 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.798 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.798 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.798 DEBUG:boto:Data: 2024-03-04T16:25:35.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.798 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.798 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.798 DEBUG:boto:Params: {} 2024-03-04T16:25:35.798 DEBUG:boto:Token: None 2024-03-04T16:25:35.799 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l6pEhBX7F4AaVa0wjjSWQOJm1BU= 2024-03-04T16:25:35.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l6pEhBX7F4AaVa0wjjSWQOJm1BU='} 2024-03-04T16:25:35.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-amz-request-id', 'tx00000a6b37bc388398931-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.811 DEBUG:rgw_multi.tests:version1 id=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:25:35.812 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.812 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.812 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.812 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.812 DEBUG:boto:Data: 2024-03-04T16:25:35.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.812 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.812 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.812 DEBUG:boto:Params: {} 2024-03-04T16:25:35.813 DEBUG:boto:Token: None 2024-03-04T16:25:35.813 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.813 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Gn+mij2s9GFgBqnLgc+sUgwl+lw= 2024-03-04T16:25:35.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Gn+mij2s9GFgBqnLgc+sUgwl+lw='} 2024-03-04T16:25:35.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC'), ('x-amz-request-id', 'tx000006795a37f03ecf234-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.823 DEBUG:rgw_multi.tests:version2 id=Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC 2024-03-04T16:25:35.824 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.824 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.824 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC 2024-03-04T16:25:35.824 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC 2024-03-04T16:25:35.824 DEBUG:boto:Method: DELETE 2024-03-04T16:25:35.824 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC 2024-03-04T16:25:35.824 DEBUG:boto:Data: 2024-03-04T16:25:35.824 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.824 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.824 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.824 DEBUG:boto:Params: {} 2024-03-04T16:25:35.825 DEBUG:boto:Token: None 2024-03-04T16:25:35.825 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone1?versionId=Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC 2024-03-04T16:25:35.825 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I+LF3HaULFSR/OAXDS56wQZXE3w= 2024-03-04T16:25:35.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I+LF3HaULFSR/OAXDS56wQZXE3w=', 'Content-Length': '0'} 2024-03-04T16:25:35.835 DEBUG:boto:Response headers: [('x-amz-version-id', 'Cr.cjJMe04vyLiVmGy.q7yNI0Ot4mCC'), ('x-amz-request-id', 'tx00000cefcd7aeb77808be-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.836 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.836 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.836 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.836 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.836 DEBUG:boto:Data: 2024-03-04T16:25:35.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.836 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.836 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.837 DEBUG:boto:Params: {} 2024-03-04T16:25:35.837 DEBUG:boto:Token: None 2024-03-04T16:25:35.837 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:b84/y+goFVOSEAAQz2Q9iCEXK70= 2024-03-04T16:25:35.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b84/y+goFVOSEAAQz2Q9iCEXK70='} 2024-03-04T16:25:35.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW'), ('x-amz-request-id', 'tx000006ed0471e9c9ae439-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.881 DEBUG:rgw_multi.tests:version3 id=UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW 2024-03-04T16:25:35.881 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.881 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:25:35.881 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW 2024-03-04T16:25:35.881 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW 2024-03-04T16:25:35.881 DEBUG:boto:Method: DELETE 2024-03-04T16:25:35.882 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW 2024-03-04T16:25:35.882 DEBUG:boto:Data: 2024-03-04T16:25:35.882 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.882 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:35.882 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.882 DEBUG:boto:Params: {} 2024-03-04T16:25:35.882 DEBUG:boto:Token: None 2024-03-04T16:25:35.883 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone1?versionId=UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW 2024-03-04T16:25:35.883 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:85HQhTgdFxA7sAVIZh/p9Bfosu0= 2024-03-04T16:25:35.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:85HQhTgdFxA7sAVIZh/p9Bfosu0=', 'Content-Length': '0'} 2024-03-04T16:25:35.893 DEBUG:boto:Response headers: [('x-amz-version-id', 'UIf-AIZ14vqNdDV4Cy3j2wjPVbBtqLW'), ('x-amz-request-id', 'tx000007d08e6c232aa0438-0065e5f5ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.894 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:25:35.894 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:25:35.894 DEBUG:boto:Method: HEAD 2024-03-04T16:25:35.894 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:25:35.894 DEBUG:boto:Data: 2024-03-04T16:25:35.894 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.894 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.894 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.894 DEBUG:boto:Params: {} 2024-03-04T16:25:35.894 DEBUG:boto:Token: None 2024-03-04T16:25:35.895 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/ 2024-03-04T16:25:35.895 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU= 2024-03-04T16:25:35.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU=', 'Content-Length': '0'} 2024-03-04T16:25:35.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3ece5eabc258345-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.898 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.898 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.898 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.898 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.898 DEBUG:boto:Data: 2024-03-04T16:25:35.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.899 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.899 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.899 DEBUG:boto:Params: {} 2024-03-04T16:25:35.899 DEBUG:boto:Token: None 2024-03-04T16:25:35.899 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.899 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9CmwzNgiiup2hMvTU2PNOs9YAic= 2024-03-04T16:25:35.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9CmwzNgiiup2hMvTU2PNOs9YAic='} 2024-03-04T16:25:35.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-amz-request-id', 'tx000007920d3b8cfd6a2f7-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.915 DEBUG:rgw_multi.tests:version1 id=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:25:35.915 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.915 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.915 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.915 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.915 DEBUG:boto:Data: 2024-03-04T16:25:35.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.916 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.916 DEBUG:boto:Params: {} 2024-03-04T16:25:35.916 DEBUG:boto:Token: None 2024-03-04T16:25:35.916 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.916 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4XdRXVDMwov48SXOMTRKaTbc2Bw= 2024-03-04T16:25:35.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4XdRXVDMwov48SXOMTRKaTbc2Bw='} 2024-03-04T16:25:35.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy'), ('x-amz-request-id', 'tx0000044eacdb393239dac-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.961 DEBUG:rgw_multi.tests:version2 id=xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy 2024-03-04T16:25:35.961 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.961 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.961 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy 2024-03-04T16:25:35.961 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy 2024-03-04T16:25:35.961 DEBUG:boto:Method: DELETE 2024-03-04T16:25:35.961 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy 2024-03-04T16:25:35.961 DEBUG:boto:Data: 2024-03-04T16:25:35.961 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.961 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.961 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.962 DEBUG:boto:Params: {} 2024-03-04T16:25:35.962 DEBUG:boto:Token: None 2024-03-04T16:25:35.962 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone2?versionId=xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy 2024-03-04T16:25:35.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hSdu14cuatIbni1ONIBVzDXYvS4= 2024-03-04T16:25:35.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hSdu14cuatIbni1ONIBVzDXYvS4=', 'Content-Length': '0'} 2024-03-04T16:25:35.972 DEBUG:boto:Response headers: [('x-amz-version-id', 'xs2GsDQZcPRnguJvfiWDmCy9cDd5Diy'), ('x-amz-request-id', 'tx000003492464e8c433769-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.973 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.973 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.973 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.973 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.973 DEBUG:boto:Data: 2024-03-04T16:25:35.973 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:35.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.973 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.973 DEBUG:boto:Params: {} 2024-03-04T16:25:35.973 DEBUG:boto:Token: None 2024-03-04T16:25:35.974 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hFa+rFp9/hyk5IKwpNUt4NYceIk= 2024-03-04T16:25:35.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hFa+rFp9/hyk5IKwpNUt4NYceIk='} 2024-03-04T16:25:35.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6'), ('x-amz-request-id', 'tx00000902f0654bdd7715c-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.983 DEBUG:rgw_multi.tests:version3 id=prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6 2024-03-04T16:25:35.984 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.984 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:25:35.984 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6 2024-03-04T16:25:35.984 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6 2024-03-04T16:25:35.984 DEBUG:boto:Method: DELETE 2024-03-04T16:25:35.984 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6 2024-03-04T16:25:35.984 DEBUG:boto:Data: 2024-03-04T16:25:35.984 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.984 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:35.984 DEBUG:boto:Port: 8000 2024-03-04T16:25:35.985 DEBUG:boto:Params: {} 2024-03-04T16:25:35.985 DEBUG:boto:Token: None 2024-03-04T16:25:35.985 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/obj-test-zone2?versionId=prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6 2024-03-04T16:25:35.985 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xMyfPZSRGbRNKOLjfxq3TnCrw4I= 2024-03-04T16:25:35.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xMyfPZSRGbRNKOLjfxq3TnCrw4I=', 'Content-Length': '0'} 2024-03-04T16:25:35.995 DEBUG:boto:Response headers: [('x-amz-version-id', 'prh2wSp1XVwzeR5ZFIGPdvt.wwuxrk6'), ('x-amz-request-id', 'tx00000ba0fdd4866d4dc7f-0065e5f5ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.995 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:25:35.995 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:25:35.995 DEBUG:boto:Method: HEAD 2024-03-04T16:25:35.995 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:25:35.995 DEBUG:boto:Data: 2024-03-04T16:25:35.995 DEBUG:boto:Headers: {} 2024-03-04T16:25:35.995 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:35.996 DEBUG:boto:Port: 8001 2024-03-04T16:25:35.996 DEBUG:boto:Params: {} 2024-03-04T16:25:35.996 DEBUG:boto:Token: None 2024-03-04T16:25:35.996 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:35 GMT /sbnvju-19/ 2024-03-04T16:25:35.996 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU= 2024-03-04T16:25:35.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:35 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uVtJa2T0VvYihoH2qgl/DemdXuU=', 'Content-Length': '0'} 2024-03-04T16:25:35.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f73dbfec11130df5-0065e5f5ff-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:35.999 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:35.999 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:35.999 DEBUG:boto:Method: PUT 2024-03-04T16:25:35.999 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:25:35.999 DEBUG:boto:Data: 2024-03-04T16:25:35.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.000 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.000 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.000 DEBUG:boto:Params: {} 2024-03-04T16:25:36.000 DEBUG:boto:Token: None 2024-03-04T16:25:36.000 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.000 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d5S+b3hJDBB1JTIr7NsO5iFFd/k= 2024-03-04T16:25:36.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d5S+b3hJDBB1JTIr7NsO5iFFd/k='} 2024-03-04T16:25:36.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-amz-request-id', 'tx00000e7b832093cdcd16b-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.013 DEBUG:rgw_multi.tests:version1 id=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:25:36.013 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.013 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.013 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.013 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.013 DEBUG:boto:Data: 2024-03-04T16:25:36.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.014 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.014 DEBUG:boto:Params: {} 2024-03-04T16:25:36.014 DEBUG:boto:Token: None 2024-03-04T16:25:36.014 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u9bbmq2FOyUn118fqH0Ns+7zCLs= 2024-03-04T16:25:36.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u9bbmq2FOyUn118fqH0Ns+7zCLs='} 2024-03-04T16:25:36.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0lJTUetnkpOJofOPc-sZdAN.1xKj35N'), ('x-amz-request-id', 'tx0000090db45a89409bc55-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.025 DEBUG:rgw_multi.tests:version2 id=0lJTUetnkpOJofOPc-sZdAN.1xKj35N 2024-03-04T16:25:36.025 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.025 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.025 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=0lJTUetnkpOJofOPc-sZdAN.1xKj35N 2024-03-04T16:25:36.025 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=0lJTUetnkpOJofOPc-sZdAN.1xKj35N 2024-03-04T16:25:36.025 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.025 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=0lJTUetnkpOJofOPc-sZdAN.1xKj35N 2024-03-04T16:25:36.025 DEBUG:boto:Data: 2024-03-04T16:25:36.026 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.026 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.026 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.026 DEBUG:boto:Params: {} 2024-03-04T16:25:36.026 DEBUG:boto:Token: None 2024-03-04T16:25:36.026 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-19/obj-test-zone3?versionId=0lJTUetnkpOJofOPc-sZdAN.1xKj35N 2024-03-04T16:25:36.026 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vPVfObwJPSIBaHGyOqo70tRdUpk= 2024-03-04T16:25:36.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vPVfObwJPSIBaHGyOqo70tRdUpk=', 'Content-Length': '0'} 2024-03-04T16:25:36.036 DEBUG:boto:Response headers: [('x-amz-version-id', '0lJTUetnkpOJofOPc-sZdAN.1xKj35N'), ('x-amz-request-id', 'tx00000831b903280c3d81f-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.036 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.036 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.037 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.037 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.037 DEBUG:boto:Data: 2024-03-04T16:25:36.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.037 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.037 DEBUG:boto:Params: {} 2024-03-04T16:25:36.037 DEBUG:boto:Token: None 2024-03-04T16:25:36.037 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.037 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a9OYuPPc4hMM3/K0/jkGmMyl8fI= 2024-03-04T16:25:36.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a9OYuPPc4hMM3/K0/jkGmMyl8fI='} 2024-03-04T16:25:36.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3AhaGKSHUHefPdnokufPpENtP2fyG8T'), ('x-amz-request-id', 'tx00000b7a272e535c37064-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.081 DEBUG:rgw_multi.tests:version3 id=3AhaGKSHUHefPdnokufPpENtP2fyG8T 2024-03-04T16:25:36.081 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.081 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:25:36.081 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=3AhaGKSHUHefPdnokufPpENtP2fyG8T 2024-03-04T16:25:36.081 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=3AhaGKSHUHefPdnokufPpENtP2fyG8T 2024-03-04T16:25:36.081 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.081 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=3AhaGKSHUHefPdnokufPpENtP2fyG8T 2024-03-04T16:25:36.081 DEBUG:boto:Data: 2024-03-04T16:25:36.081 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.081 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.081 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.082 DEBUG:boto:Params: {} 2024-03-04T16:25:36.082 DEBUG:boto:Token: None 2024-03-04T16:25:36.082 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-19/obj-test-zone3?versionId=3AhaGKSHUHefPdnokufPpENtP2fyG8T 2024-03-04T16:25:36.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okb9ExkG/GXW3CNBHzaF1+vGc1A= 2024-03-04T16:25:36.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okb9ExkG/GXW3CNBHzaF1+vGc1A=', 'Content-Length': '0'} 2024-03-04T16:25:36.092 DEBUG:boto:Response headers: [('x-amz-version-id', '3AhaGKSHUHefPdnokufPpENtP2fyG8T'), ('x-amz-request-id', 'tx00000cf7c293802fef780-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.092 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:25:36.092 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:25:36.092 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.092 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:25:36.092 DEBUG:boto:Data: 2024-03-04T16:25:36.092 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.093 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.093 DEBUG:boto:Params: {} 2024-03-04T16:25:36.093 DEBUG:boto:Token: None 2024-03-04T16:25:36.093 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/ 2024-03-04T16:25:36.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE= 2024-03-04T16:25:36.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE=', 'Content-Length': '0'} 2024-03-04T16:25:36.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afe96e914d112c30-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.096 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.096 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.096 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.096 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.096 DEBUG:boto:Data: 2024-03-04T16:25:36.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.096 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.096 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.096 DEBUG:boto:Params: {} 2024-03-04T16:25:36.097 DEBUG:boto:Token: None 2024-03-04T16:25:36.097 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.097 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Hiim7SzjeEphkVkeW/LZI/HwWc= 2024-03-04T16:25:36.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Hiim7SzjeEphkVkeW/LZI/HwWc='} 2024-03-04T16:25:36.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-amz-request-id', 'tx00000e6e09b7d2c81a816-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.107 DEBUG:rgw_multi.tests:version1 id=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:25:36.108 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.108 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.108 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.108 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.108 DEBUG:boto:Data: 2024-03-04T16:25:36.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.108 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.108 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.108 DEBUG:boto:Params: {} 2024-03-04T16:25:36.108 DEBUG:boto:Token: None 2024-03-04T16:25:36.109 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.109 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KNcCve9AfigaCUu7HABIUvYgssw= 2024-03-04T16:25:36.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KNcCve9AfigaCUu7HABIUvYgssw='} 2024-03-04T16:25:36.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sEGl4Q9geogPRllRC-tEk509gZOd74G'), ('x-amz-request-id', 'tx00000e93506590b972212-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.118 DEBUG:rgw_multi.tests:version2 id=sEGl4Q9geogPRllRC-tEk509gZOd74G 2024-03-04T16:25:36.118 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.118 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.118 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=sEGl4Q9geogPRllRC-tEk509gZOd74G 2024-03-04T16:25:36.119 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=sEGl4Q9geogPRllRC-tEk509gZOd74G 2024-03-04T16:25:36.119 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.119 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=sEGl4Q9geogPRllRC-tEk509gZOd74G 2024-03-04T16:25:36.119 DEBUG:boto:Data: 2024-03-04T16:25:36.119 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.119 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.119 DEBUG:boto:Params: {} 2024-03-04T16:25:36.119 DEBUG:boto:Token: None 2024-03-04T16:25:36.119 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone1?versionId=sEGl4Q9geogPRllRC-tEk509gZOd74G 2024-03-04T16:25:36.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EgpknMFQCcGgySF4Q3Q25DWrIEY= 2024-03-04T16:25:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EgpknMFQCcGgySF4Q3Q25DWrIEY=', 'Content-Length': '0'} 2024-03-04T16:25:36.129 DEBUG:boto:Response headers: [('x-amz-version-id', 'sEGl4Q9geogPRllRC-tEk509gZOd74G'), ('x-amz-request-id', 'tx00000bdf27eab7eed3019-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.129 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.129 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.129 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.130 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.130 DEBUG:boto:Data: 2024-03-04T16:25:36.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.130 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.130 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.130 DEBUG:boto:Params: {} 2024-03-04T16:25:36.130 DEBUG:boto:Token: None 2024-03-04T16:25:36.130 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VCiauYvnVWNj+5RXdQz35ygYSus= 2024-03-04T16:25:36.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VCiauYvnVWNj+5RXdQz35ygYSus='} 2024-03-04T16:25:36.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kEEYc8izI4KuHfkTJAs.bpywNQTq7w-'), ('x-amz-request-id', 'tx00000b5cbbd6fced7da0a-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.177 DEBUG:rgw_multi.tests:version3 id=kEEYc8izI4KuHfkTJAs.bpywNQTq7w- 2024-03-04T16:25:36.177 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.177 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:25:36.177 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=kEEYc8izI4KuHfkTJAs.bpywNQTq7w- 2024-03-04T16:25:36.177 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=kEEYc8izI4KuHfkTJAs.bpywNQTq7w- 2024-03-04T16:25:36.177 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.177 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=kEEYc8izI4KuHfkTJAs.bpywNQTq7w- 2024-03-04T16:25:36.177 DEBUG:boto:Data: 2024-03-04T16:25:36.177 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.177 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.178 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.178 DEBUG:boto:Params: {} 2024-03-04T16:25:36.178 DEBUG:boto:Token: None 2024-03-04T16:25:36.178 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone1?versionId=kEEYc8izI4KuHfkTJAs.bpywNQTq7w- 2024-03-04T16:25:36.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vd+nVcJ96npmE8dVrmpKKwYebks= 2024-03-04T16:25:36.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vd+nVcJ96npmE8dVrmpKKwYebks=', 'Content-Length': '0'} 2024-03-04T16:25:36.187 DEBUG:boto:Response headers: [('x-amz-version-id', 'kEEYc8izI4KuHfkTJAs.bpywNQTq7w-'), ('x-amz-request-id', 'tx000001baef3c6875168ec-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.189 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:25:36.189 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:25:36.189 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.189 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:25:36.189 DEBUG:boto:Data: 2024-03-04T16:25:36.189 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.189 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.189 DEBUG:boto:Params: {} 2024-03-04T16:25:36.189 DEBUG:boto:Token: None 2024-03-04T16:25:36.190 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/ 2024-03-04T16:25:36.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE= 2024-03-04T16:25:36.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE=', 'Content-Length': '0'} 2024-03-04T16:25:36.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018e75774149ffeb9-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.192 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.192 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.193 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.193 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.193 DEBUG:boto:Data: 2024-03-04T16:25:36.193 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.193 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.193 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.193 DEBUG:boto:Params: {} 2024-03-04T16:25:36.193 DEBUG:boto:Token: None 2024-03-04T16:25:36.193 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3jnzdDX9BOa7tMXELZ+72eS3iRo= 2024-03-04T16:25:36.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3jnzdDX9BOa7tMXELZ+72eS3iRo='} 2024-03-04T16:25:36.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-amz-request-id', 'tx00000dcbc37ab1ee84cfb-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.206 DEBUG:rgw_multi.tests:version1 id=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:25:36.206 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.206 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.206 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.206 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.206 DEBUG:boto:Data: 2024-03-04T16:25:36.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.206 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.207 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.207 DEBUG:boto:Params: {} 2024-03-04T16:25:36.207 DEBUG:boto:Token: None 2024-03-04T16:25:36.207 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.207 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I2BVPLFm2jACd38fXa/IqukawyY= 2024-03-04T16:25:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I2BVPLFm2jACd38fXa/IqukawyY='} 2024-03-04T16:25:36.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mwYu1knMkLSCMeqOlMfFZds4sBWbe2c'), ('x-amz-request-id', 'tx00000cc7f5313c4546062-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.218 DEBUG:rgw_multi.tests:version2 id=mwYu1knMkLSCMeqOlMfFZds4sBWbe2c 2024-03-04T16:25:36.218 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.218 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.218 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=mwYu1knMkLSCMeqOlMfFZds4sBWbe2c 2024-03-04T16:25:36.218 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=mwYu1knMkLSCMeqOlMfFZds4sBWbe2c 2024-03-04T16:25:36.218 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.218 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=mwYu1knMkLSCMeqOlMfFZds4sBWbe2c 2024-03-04T16:25:36.219 DEBUG:boto:Data: 2024-03-04T16:25:36.219 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.219 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.219 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.219 DEBUG:boto:Params: {} 2024-03-04T16:25:36.219 DEBUG:boto:Token: None 2024-03-04T16:25:36.219 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone2?versionId=mwYu1knMkLSCMeqOlMfFZds4sBWbe2c 2024-03-04T16:25:36.219 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MLuRTwmWjrA/SeTaaRic5OAct4g= 2024-03-04T16:25:36.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MLuRTwmWjrA/SeTaaRic5OAct4g=', 'Content-Length': '0'} 2024-03-04T16:25:36.229 DEBUG:boto:Response headers: [('x-amz-version-id', 'mwYu1knMkLSCMeqOlMfFZds4sBWbe2c'), ('x-amz-request-id', 'tx00000a615dd61329d5896-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.229 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.229 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.230 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.230 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.230 DEBUG:boto:Data: 2024-03-04T16:25:36.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.230 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.230 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.230 DEBUG:boto:Params: {} 2024-03-04T16:25:36.230 DEBUG:boto:Token: None 2024-03-04T16:25:36.230 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VuzvHk0TSgIQ7jz+wpqOof3x9C0= 2024-03-04T16:25:36.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VuzvHk0TSgIQ7jz+wpqOof3x9C0='} 2024-03-04T16:25:36.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zcBqDP.pTZUbpiK5vSxK.zYK9310cCE'), ('x-amz-request-id', 'tx00000fdd4a0382d771055-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.272 DEBUG:rgw_multi.tests:version3 id=zcBqDP.pTZUbpiK5vSxK.zYK9310cCE 2024-03-04T16:25:36.273 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.273 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:25:36.273 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=zcBqDP.pTZUbpiK5vSxK.zYK9310cCE 2024-03-04T16:25:36.273 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=zcBqDP.pTZUbpiK5vSxK.zYK9310cCE 2024-03-04T16:25:36.273 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.273 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=zcBqDP.pTZUbpiK5vSxK.zYK9310cCE 2024-03-04T16:25:36.273 DEBUG:boto:Data: 2024-03-04T16:25:36.273 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.273 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.273 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.273 DEBUG:boto:Params: {} 2024-03-04T16:25:36.274 DEBUG:boto:Token: None 2024-03-04T16:25:36.274 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone2?versionId=zcBqDP.pTZUbpiK5vSxK.zYK9310cCE 2024-03-04T16:25:36.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/637oPeD4ClziIl/KvgS1TgoH6w= 2024-03-04T16:25:36.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/637oPeD4ClziIl/KvgS1TgoH6w=', 'Content-Length': '0'} 2024-03-04T16:25:36.286 DEBUG:boto:Response headers: [('x-amz-version-id', 'zcBqDP.pTZUbpiK5vSxK.zYK9310cCE'), ('x-amz-request-id', 'tx00000880937b78a2f589f-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.287 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:25:36.287 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:25:36.287 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.287 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:25:36.287 DEBUG:boto:Data: 2024-03-04T16:25:36.287 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.287 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.287 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.287 DEBUG:boto:Params: {} 2024-03-04T16:25:36.288 DEBUG:boto:Token: None 2024-03-04T16:25:36.288 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/ 2024-03-04T16:25:36.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE= 2024-03-04T16:25:36.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M65/QC/lPrpaMMOn4dut9+H57RE=', 'Content-Length': '0'} 2024-03-04T16:25:36.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef17b71cbf924b1e-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.291 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.291 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.291 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.291 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.291 DEBUG:boto:Data: 2024-03-04T16:25:36.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.291 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.291 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.291 DEBUG:boto:Params: {} 2024-03-04T16:25:36.291 DEBUG:boto:Token: None 2024-03-04T16:25:36.292 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.292 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c64czq3ueV0IwnWVGOtl0f5I6zQ= 2024-03-04T16:25:36.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c64czq3ueV0IwnWVGOtl0f5I6zQ='} 2024-03-04T16:25:36.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-amz-request-id', 'tx000007fdaebe87c01f618-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.301 DEBUG:rgw_multi.tests:version1 id=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:25:36.301 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.301 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.301 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.302 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.302 DEBUG:boto:Data: 2024-03-04T16:25:36.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.302 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.302 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.302 DEBUG:boto:Params: {} 2024-03-04T16:25:36.302 DEBUG:boto:Token: None 2024-03-04T16:25:36.302 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:e7TwLJS1rP/pZ4p1qzwftruYuLA= 2024-03-04T16:25:36.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:e7TwLJS1rP/pZ4p1qzwftruYuLA='} 2024-03-04T16:25:36.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NIZqpCF03Vd--h.F6n3lni.L-0JVahj'), ('x-amz-request-id', 'tx0000033ce798c21e27ee7-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.311 DEBUG:rgw_multi.tests:version2 id=NIZqpCF03Vd--h.F6n3lni.L-0JVahj 2024-03-04T16:25:36.311 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.312 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.312 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=NIZqpCF03Vd--h.F6n3lni.L-0JVahj 2024-03-04T16:25:36.312 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=NIZqpCF03Vd--h.F6n3lni.L-0JVahj 2024-03-04T16:25:36.312 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.312 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=NIZqpCF03Vd--h.F6n3lni.L-0JVahj 2024-03-04T16:25:36.312 DEBUG:boto:Data: 2024-03-04T16:25:36.312 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.312 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.312 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.312 DEBUG:boto:Params: {} 2024-03-04T16:25:36.312 DEBUG:boto:Token: None 2024-03-04T16:25:36.313 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone3?versionId=NIZqpCF03Vd--h.F6n3lni.L-0JVahj 2024-03-04T16:25:36.313 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9vbcjmjD9dbiJe7vWG5CaUuMU3g= 2024-03-04T16:25:36.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9vbcjmjD9dbiJe7vWG5CaUuMU3g=', 'Content-Length': '0'} 2024-03-04T16:25:36.321 DEBUG:boto:Response headers: [('x-amz-version-id', 'NIZqpCF03Vd--h.F6n3lni.L-0JVahj'), ('x-amz-request-id', 'tx000003df2c63da39fcac3-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.322 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.322 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.322 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.322 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.322 DEBUG:boto:Data: 2024-03-04T16:25:36.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.322 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.322 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.322 DEBUG:boto:Params: {} 2024-03-04T16:25:36.323 DEBUG:boto:Token: None 2024-03-04T16:25:36.323 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uKTUAO3kX9sOHwxrNfdDPvfQU9s= 2024-03-04T16:25:36.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uKTUAO3kX9sOHwxrNfdDPvfQU9s='} 2024-03-04T16:25:36.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hKPlZvDpl4c8n15SbCjHB4OAigRJX4j'), ('x-amz-request-id', 'tx00000c10548947fae4ee7-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.369 DEBUG:rgw_multi.tests:version3 id=hKPlZvDpl4c8n15SbCjHB4OAigRJX4j 2024-03-04T16:25:36.369 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.369 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:25:36.369 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=hKPlZvDpl4c8n15SbCjHB4OAigRJX4j 2024-03-04T16:25:36.369 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=hKPlZvDpl4c8n15SbCjHB4OAigRJX4j 2024-03-04T16:25:36.369 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.369 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=hKPlZvDpl4c8n15SbCjHB4OAigRJX4j 2024-03-04T16:25:36.369 DEBUG:boto:Data: 2024-03-04T16:25:36.369 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.370 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.370 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.370 DEBUG:boto:Params: {} 2024-03-04T16:25:36.370 DEBUG:boto:Token: None 2024-03-04T16:25:36.370 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-20/obj-test-zone3?versionId=hKPlZvDpl4c8n15SbCjHB4OAigRJX4j 2024-03-04T16:25:36.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lwjteLmsiwLkyOkH50fmyXfDnKk= 2024-03-04T16:25:36.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lwjteLmsiwLkyOkH50fmyXfDnKk=', 'Content-Length': '0'} 2024-03-04T16:25:36.380 DEBUG:boto:Response headers: [('x-amz-version-id', 'hKPlZvDpl4c8n15SbCjHB4OAigRJX4j'), ('x-amz-request-id', 'tx00000428cafd43aa59a23-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.380 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:25:36.380 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:25:36.380 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.380 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:25:36.380 DEBUG:boto:Data: 2024-03-04T16:25:36.381 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.381 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.381 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.381 DEBUG:boto:Params: {} 2024-03-04T16:25:36.381 DEBUG:boto:Token: None 2024-03-04T16:25:36.381 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/ 2024-03-04T16:25:36.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U= 2024-03-04T16:25:36.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U=', 'Content-Length': '0'} 2024-03-04T16:25:36.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092587a51942d71b6-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.384 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.384 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.384 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.384 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.384 DEBUG:boto:Data: 2024-03-04T16:25:36.384 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.384 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.384 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.385 DEBUG:boto:Params: {} 2024-03-04T16:25:36.385 DEBUG:boto:Token: None 2024-03-04T16:25:36.385 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1j5mYVyO0IrKTFhSzH856IvdstI= 2024-03-04T16:25:36.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1j5mYVyO0IrKTFhSzH856IvdstI='} 2024-03-04T16:25:36.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-amz-request-id', 'tx000009f6f6c201a8c51fa-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.395 DEBUG:rgw_multi.tests:version1 id=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:25:36.395 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.395 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.395 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.396 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.396 DEBUG:boto:Data: 2024-03-04T16:25:36.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.396 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.396 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.396 DEBUG:boto:Params: {} 2024-03-04T16:25:36.396 DEBUG:boto:Token: None 2024-03-04T16:25:36.396 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.396 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GtgmJwGNYymck0OQkDf1inzFTvI= 2024-03-04T16:25:36.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GtgmJwGNYymck0OQkDf1inzFTvI='} 2024-03-04T16:25:36.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'toknXC34s5Ff00VbXG2W7Tp3NsbTO4s'), ('x-amz-request-id', 'tx00000a546fc5099a4d77d-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.405 DEBUG:rgw_multi.tests:version2 id=toknXC34s5Ff00VbXG2W7Tp3NsbTO4s 2024-03-04T16:25:36.405 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.405 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.405 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=toknXC34s5Ff00VbXG2W7Tp3NsbTO4s 2024-03-04T16:25:36.406 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=toknXC34s5Ff00VbXG2W7Tp3NsbTO4s 2024-03-04T16:25:36.406 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.406 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=toknXC34s5Ff00VbXG2W7Tp3NsbTO4s 2024-03-04T16:25:36.406 DEBUG:boto:Data: 2024-03-04T16:25:36.406 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.406 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.406 DEBUG:boto:Params: {} 2024-03-04T16:25:36.406 DEBUG:boto:Token: None 2024-03-04T16:25:36.406 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone1?versionId=toknXC34s5Ff00VbXG2W7Tp3NsbTO4s 2024-03-04T16:25:36.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kxIoWd2TuAAeKzEhoqJCzECMViQ= 2024-03-04T16:25:36.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kxIoWd2TuAAeKzEhoqJCzECMViQ=', 'Content-Length': '0'} 2024-03-04T16:25:36.415 DEBUG:boto:Response headers: [('x-amz-version-id', 'toknXC34s5Ff00VbXG2W7Tp3NsbTO4s'), ('x-amz-request-id', 'tx00000363e37c58486ea86-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.415 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.415 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.415 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.416 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.416 DEBUG:boto:Data: 2024-03-04T16:25:36.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.416 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.416 DEBUG:boto:Params: {} 2024-03-04T16:25:36.416 DEBUG:boto:Token: None 2024-03-04T16:25:36.416 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.416 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FU/MVTVNab2J+SEMvnAyexMo9tE= 2024-03-04T16:25:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FU/MVTVNab2J+SEMvnAyexMo9tE='} 2024-03-04T16:25:36.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1'), ('x-amz-request-id', 'tx000008f8ead768204268c-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.460 DEBUG:rgw_multi.tests:version3 id=9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1 2024-03-04T16:25:36.461 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.461 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:25:36.461 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1 2024-03-04T16:25:36.461 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1 2024-03-04T16:25:36.461 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.461 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1 2024-03-04T16:25:36.461 DEBUG:boto:Data: 2024-03-04T16:25:36.461 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.461 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:25:36.461 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.461 DEBUG:boto:Params: {} 2024-03-04T16:25:36.462 DEBUG:boto:Token: None 2024-03-04T16:25:36.462 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone1?versionId=9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1 2024-03-04T16:25:36.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QNFTXBStXiG1+I3QriiStQAXc/g= 2024-03-04T16:25:36.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QNFTXBStXiG1+I3QriiStQAXc/g=', 'Content-Length': '0'} 2024-03-04T16:25:36.471 DEBUG:boto:Response headers: [('x-amz-version-id', '9NKb4yiEyuz8KbgRexLHoN-k-P18Oe1'), ('x-amz-request-id', 'tx0000057a144295a8050fa-0065e5f600-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.471 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:25:36.472 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:25:36.472 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.472 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:25:36.472 DEBUG:boto:Data: 2024-03-04T16:25:36.472 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.472 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.472 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.472 DEBUG:boto:Params: {} 2024-03-04T16:25:36.472 DEBUG:boto:Token: None 2024-03-04T16:25:36.473 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/ 2024-03-04T16:25:36.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U= 2024-03-04T16:25:36.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U=', 'Content-Length': '0'} 2024-03-04T16:25:36.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2bcba509de8c77a-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.475 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.476 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.476 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.476 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.476 DEBUG:boto:Data: 2024-03-04T16:25:36.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.476 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.476 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.476 DEBUG:boto:Params: {} 2024-03-04T16:25:36.476 DEBUG:boto:Token: None 2024-03-04T16:25:36.476 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FyeHWRLRg2IYnvg4CrOS2N2Ehrs= 2024-03-04T16:25:36.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FyeHWRLRg2IYnvg4CrOS2N2Ehrs='} 2024-03-04T16:25:36.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-amz-request-id', 'tx0000033ce78db2187b8bd-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.488 DEBUG:rgw_multi.tests:version1 id=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:25:36.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.488 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.489 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.489 DEBUG:boto:Data: 2024-03-04T16:25:36.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.489 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.489 DEBUG:boto:Params: {} 2024-03-04T16:25:36.489 DEBUG:boto:Token: None 2024-03-04T16:25:36.489 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bMSXEVOC/oryQWYttJbeoF8CAJw= 2024-03-04T16:25:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bMSXEVOC/oryQWYttJbeoF8CAJw='} 2024-03-04T16:25:36.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7'), ('x-amz-request-id', 'tx0000074f19e0671b6c193-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.499 DEBUG:rgw_multi.tests:version2 id=Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7 2024-03-04T16:25:36.499 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.500 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.500 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7 2024-03-04T16:25:36.500 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7 2024-03-04T16:25:36.500 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.500 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7 2024-03-04T16:25:36.500 DEBUG:boto:Data: 2024-03-04T16:25:36.500 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.500 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.500 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.500 DEBUG:boto:Params: {} 2024-03-04T16:25:36.501 DEBUG:boto:Token: None 2024-03-04T16:25:36.501 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone2?versionId=Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7 2024-03-04T16:25:36.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OBSWG1KSWLGzdd3y9RDaVR9GAXY= 2024-03-04T16:25:36.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OBSWG1KSWLGzdd3y9RDaVR9GAXY=', 'Content-Length': '0'} 2024-03-04T16:25:36.510 DEBUG:boto:Response headers: [('x-amz-version-id', 'Ilx2464mWgUJ9nvv9M.Iu8SsRnGfTM7'), ('x-amz-request-id', 'tx0000036179ed0b9199632-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.510 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.510 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.510 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.510 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.510 DEBUG:boto:Data: 2024-03-04T16:25:36.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.511 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.511 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.511 DEBUG:boto:Params: {} 2024-03-04T16:25:36.511 DEBUG:boto:Token: None 2024-03-04T16:25:36.511 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVfSUrRWmnifXqmqQZh+vv+7ybs= 2024-03-04T16:25:36.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVfSUrRWmnifXqmqQZh+vv+7ybs='} 2024-03-04T16:25:36.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4'), ('x-amz-request-id', 'tx00000f1098d6b9024acf3-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.557 DEBUG:rgw_multi.tests:version3 id=YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4 2024-03-04T16:25:36.557 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.557 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:25:36.557 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4 2024-03-04T16:25:36.557 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4 2024-03-04T16:25:36.557 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.557 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4 2024-03-04T16:25:36.557 DEBUG:boto:Data: 2024-03-04T16:25:36.557 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.558 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:25:36.558 DEBUG:boto:Port: 8000 2024-03-04T16:25:36.558 DEBUG:boto:Params: {} 2024-03-04T16:25:36.558 DEBUG:boto:Token: None 2024-03-04T16:25:36.558 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone2?versionId=YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4 2024-03-04T16:25:36.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zpspTaJG7KmCaYrfYDolvBGkUmw= 2024-03-04T16:25:36.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zpspTaJG7KmCaYrfYDolvBGkUmw=', 'Content-Length': '0'} 2024-03-04T16:25:36.567 DEBUG:boto:Response headers: [('x-amz-version-id', 'YAbo0Q9lemTKBzce.dr3OTl.AL-6kk4'), ('x-amz-request-id', 'tx00000295d6bd843316229-0065e5f600-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.568 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:25:36.568 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:25:36.568 DEBUG:boto:Method: HEAD 2024-03-04T16:25:36.568 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:25:36.568 DEBUG:boto:Data: 2024-03-04T16:25:36.568 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.568 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.568 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.568 DEBUG:boto:Params: {} 2024-03-04T16:25:36.569 DEBUG:boto:Token: None 2024-03-04T16:25:36.569 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/ 2024-03-04T16:25:36.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U= 2024-03-04T16:25:36.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cr1Ud1S+rLNRpKpHObvEm3Yke+U=', 'Content-Length': '0'} 2024-03-04T16:25:36.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000670166cefe0556bc-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.573 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.573 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.573 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.573 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.573 DEBUG:boto:Data: 2024-03-04T16:25:36.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.573 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.573 DEBUG:boto:Params: {} 2024-03-04T16:25:36.574 DEBUG:boto:Token: None 2024-03-04T16:25:36.574 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:b8arz7Bo6Prd8IDjqS5ciGilcTo= 2024-03-04T16:25:36.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b8arz7Bo6Prd8IDjqS5ciGilcTo='} 2024-03-04T16:25:36.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-amz-request-id', 'tx00000cac98669be0c377d-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.583 DEBUG:rgw_multi.tests:version1 id=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:25:36.584 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.584 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.584 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.584 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.584 DEBUG:boto:Data: 2024-03-04T16:25:36.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.584 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.584 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.584 DEBUG:boto:Params: {} 2024-03-04T16:25:36.585 DEBUG:boto:Token: None 2024-03-04T16:25:36.585 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.585 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LB604/glDrNDhoR/IFUrQuDUCVw= 2024-03-04T16:25:36.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LB604/glDrNDhoR/IFUrQuDUCVw='} 2024-03-04T16:25:36.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir'), ('x-amz-request-id', 'tx000006b5a783453ab5309-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.633 DEBUG:rgw_multi.tests:version2 id=LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir 2024-03-04T16:25:36.633 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.633 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.633 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir 2024-03-04T16:25:36.633 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir 2024-03-04T16:25:36.633 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.633 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir 2024-03-04T16:25:36.633 DEBUG:boto:Data: 2024-03-04T16:25:36.633 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.633 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.634 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.634 DEBUG:boto:Params: {} 2024-03-04T16:25:36.634 DEBUG:boto:Token: None 2024-03-04T16:25:36.634 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone3?versionId=LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir 2024-03-04T16:25:36.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiK8uAvCvl8wrT9RlQprm9IuE64= 2024-03-04T16:25:36.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiK8uAvCvl8wrT9RlQprm9IuE64=', 'Content-Length': '0'} 2024-03-04T16:25:36.643 DEBUG:boto:Response headers: [('x-amz-version-id', 'LEPC2egtTE.KFsXD4d5lZZhaYT7Cuir'), ('x-amz-request-id', 'tx000002a04d7cc1fd4735d-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.643 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.643 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.643 DEBUG:boto:Method: PUT 2024-03-04T16:25:36.644 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.644 DEBUG:boto:Data: 2024-03-04T16:25:36.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:25:36.644 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.644 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.644 DEBUG:boto:Params: {} 2024-03-04T16:25:36.644 DEBUG:boto:Token: None 2024-03-04T16:25:36.644 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.644 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q9frbs9fGRf9rUEba68/IX/dTt0= 2024-03-04T16:25:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q9frbs9fGRf9rUEba68/IX/dTt0='} 2024-03-04T16:25:36.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pbDtA0uPT1WlzKRb06WO44oukzf1Iwl'), ('x-amz-request-id', 'tx00000cae546177927a9e6-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.653 DEBUG:rgw_multi.tests:version3 id=pbDtA0uPT1WlzKRb06WO44oukzf1Iwl 2024-03-04T16:25:36.653 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.653 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:25:36.654 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=pbDtA0uPT1WlzKRb06WO44oukzf1Iwl 2024-03-04T16:25:36.654 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=pbDtA0uPT1WlzKRb06WO44oukzf1Iwl 2024-03-04T16:25:36.654 DEBUG:boto:Method: DELETE 2024-03-04T16:25:36.654 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=pbDtA0uPT1WlzKRb06WO44oukzf1Iwl 2024-03-04T16:25:36.654 DEBUG:boto:Data: 2024-03-04T16:25:36.654 DEBUG:boto:Headers: {} 2024-03-04T16:25:36.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:25:36.654 DEBUG:boto:Port: 8001 2024-03-04T16:25:36.654 DEBUG:boto:Params: {} 2024-03-04T16:25:36.654 DEBUG:boto:Token: None 2024-03-04T16:25:36.654 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:25:36 GMT /sbnvju-21/obj-test-zone3?versionId=pbDtA0uPT1WlzKRb06WO44oukzf1Iwl 2024-03-04T16:25:36.655 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Zawpktulnfu8C2fhirK/maIMNSc= 2024-03-04T16:25:36.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:25:36 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Zawpktulnfu8C2fhirK/maIMNSc=', 'Content-Length': '0'} 2024-03-04T16:25:36.663 DEBUG:boto:Response headers: [('x-amz-version-id', 'pbDtA0uPT1WlzKRb06WO44oukzf1Iwl'), ('x-amz-request-id', 'tx00000fe16fb9ab568e292-0065e5f600-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:25:36.663 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-19 2024-03-04T16:25:36.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:36.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:36.664 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:36.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:36.772+0000 7fc32a830800 1 bucket sync caught up with source: 2024-03-04T16:25:36.772 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:36.772 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:36.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:25:36.772+0000 7fc32a830800 0 bucket checkpoint complete 2024-03-04T16:25:36.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-19 2024-03-04T16:25:36.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:36.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:36.790 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:36.846 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:36.846 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:36.895+0000 7fed18a2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:25:36.895 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:36.895 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:41.896+0000 7fed18a2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:25:41.897 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:41.897 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:46.897+0000 7fed18a2e800 1 bucket sync caught up with source: 2024-03-04T16:25:46.900 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:46.901 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000010.43.6, , , , 00000000005.53.10, , , , , ] 2024-03-04T16:25:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:46.898+0000 7fed18a2e800 0 bucket checkpoint complete 2024-03-04T16:25:46.922 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-19 2024-03-04T16:25:46.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:46.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:46.922 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:46.982 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:46.982 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:47.036+0000 7f8cf7a44800 1 waiting for incremental sync to catch up: 2024-03-04T16:25:47.036 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.42.10, , , , , ] 2024-03-04T16:25:47.036 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.21.11, , 00000000010.36.6, , 00000000003.42.10, , , , , ] 2024-03-04T16:25:52.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:52.037+0000 7f8cf7a44800 1 waiting for incremental sync to catch up: 2024-03-04T16:25:52.038 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.42.10, , , , , ] 2024-03-04T16:25:52.038 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.21.11, , 00000000010.36.6, , 00000000003.42.10, , , , , ] 2024-03-04T16:25:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:57.038+0000 7f8cf7a44800 1 bucket sync caught up with source: 2024-03-04T16:25:57.039 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.21.11, , 00000000010.36.6, , 00000000003.42.10, , , , , ] 2024-03-04T16:25:57.039 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000002.21.11, , 00000000010.36.6, , 00000000003.42.10, , , , , ] 2024-03-04T16:25:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:57.038+0000 7f8cf7a44800 0 bucket checkpoint complete 2024-03-04T16:25:57.063 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-19 2024-03-04T16:25:57.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:57.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:25:57.064 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:25:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:25:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:25:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:25:57.186+0000 7fb5ce630800 1 waiting for incremental sync to catch up: 2024-03-04T16:25:57.187 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000002.21.11, , 00000000010.36.6, , 00000000003.42.10, , , , , ] 2024-03-04T16:25:57.187 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000010.36.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:26:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:02.188+0000 7fb5ce630800 1 bucket sync caught up with source: 2024-03-04T16:26:02.188 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000010.36.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:26:02.188 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000010.36.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:26:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:02.188+0000 7fb5ce630800 0 bucket checkpoint complete 2024-03-04T16:26:02.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-19 2024-03-04T16:26:02.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:02.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:02.212 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:02.271 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:02.271 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:02.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:02.324+0000 7fc9e7811800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:02.324 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000002.15.11, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:02.324 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:07.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:07.325+0000 7fc9e7811800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:07.326 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000002.15.11, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:07.326 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:12.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:12.326+0000 7fc9e7811800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:12.327 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000002.15.11, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:12.327 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:17.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:17.327+0000 7fc9e7811800 1 bucket sync caught up with source: 2024-03-04T16:26:17.328 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:17.328 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:17.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:17.327+0000 7fc9e7811800 0 bucket checkpoint complete 2024-03-04T16:26:17.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-19 2024-03-04T16:26:17.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:17.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:17.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:17.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:17.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:17.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:17.467+0000 7f12cdb33800 1 bucket sync caught up with source: 2024-03-04T16:26:17.468 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:17.468 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000014.64.6, , , , , ] 2024-03-04T16:26:17.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:17.467+0000 7f12cdb33800 0 bucket checkpoint complete 2024-03-04T16:26:17.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone2, test-zone1} 2024-03-04T16:26:17.486 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.486 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.487 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.487 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:17.487 DEBUG:boto:Data: 2024-03-04T16:26:17.487 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.487 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.487 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.487 DEBUG:boto:Params: {} 2024-03-04T16:26:17.487 DEBUG:boto:Token: None 2024-03-04T16:26:17.488 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8nrgOjR/PsU3dElJDT3LFE+1CgY= 2024-03-04T16:26:17.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8nrgOjR/PsU3dElJDT3LFE+1CgY=', 'Content-Length': '0'} 2024-03-04T16:26:17.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a73e821554b4f80-0065e5f629-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.491 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.491 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.491 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.491 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:17.491 DEBUG:boto:Data: 2024-03-04T16:26:17.491 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.491 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.492 DEBUG:boto:Params: {} 2024-03-04T16:26:17.492 DEBUG:boto:Token: None 2024-03-04T16:26:17.492 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.492 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8nrgOjR/PsU3dElJDT3LFE+1CgY= 2024-03-04T16:26:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8nrgOjR/PsU3dElJDT3LFE+1CgY=', 'Content-Length': '0'} 2024-03-04T16:26:17.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cee76d3f3bd035ed-0065e5f629-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:17.496 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.496 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.496 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.496 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.497 DEBUG:boto:Method: GET 2024-03-04T16:26:17.497 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.497 DEBUG:boto:Data: 2024-03-04T16:26:17.497 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.497 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.497 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.497 DEBUG:boto:Params: {} 2024-03-04T16:26:17.497 DEBUG:boto:Token: None 2024-03-04T16:26:17.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/?versions 2024-03-04T16:26:17.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg= 2024-03-04T16:26:17.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg=', 'Content-Length': '0'} 2024-03-04T16:26:17.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f993b19eb9df76-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.500 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.501 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:17.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:17.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:17.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:17.501 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:17.501 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.501 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.501 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.501 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.502 DEBUG:boto:Method: GET 2024-03-04T16:26:17.502 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.502 DEBUG:boto:Data: 2024-03-04T16:26:17.502 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.502 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.502 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.502 DEBUG:boto:Params: {} 2024-03-04T16:26:17.502 DEBUG:boto:Token: None 2024-03-04T16:26:17.502 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/?versions 2024-03-04T16:26:17.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg= 2024-03-04T16:26:17.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg=', 'Content-Length': '0'} 2024-03-04T16:26:17.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000324f7b13b6f586af-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.506 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.507 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:17.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:17.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:17.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:17.507 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.507 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.507 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.507 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.507 DEBUG:boto:Method: GET 2024-03-04T16:26:17.507 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.508 DEBUG:boto:Data: 2024-03-04T16:26:17.508 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.508 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.508 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.508 DEBUG:boto:Params: {} 2024-03-04T16:26:17.508 DEBUG:boto:Token: None 2024-03-04T16:26:17.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/?versions 2024-03-04T16:26:17.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg= 2024-03-04T16:26:17.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg=', 'Content-Length': '0'} 2024-03-04T16:26:17.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f622cafdf3f27607-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.511 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.511 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.511 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.511 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.511 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.511 DEBUG:boto:Method: GET 2024-03-04T16:26:17.512 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:17.512 DEBUG:boto:Data: 2024-03-04T16:26:17.512 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.512 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.512 DEBUG:boto:Params: {} 2024-03-04T16:26:17.512 DEBUG:boto:Token: None 2024-03-04T16:26:17.512 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/?versions 2024-03-04T16:26:17.512 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg= 2024-03-04T16:26:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BNf1TKr7iBs+ktrxDn+8xaPG0Xg=', 'Content-Length': '0'} 2024-03-04T16:26:17.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e9d659964a860d-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.515 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:17.516 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.516 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.516 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.516 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.516 DEBUG:boto:Method: GET 2024-03-04T16:26:17.516 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.516 DEBUG:boto:Data: 2024-03-04T16:26:17.516 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.517 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.517 DEBUG:boto:Params: {} 2024-03-04T16:26:17.517 DEBUG:boto:Token: None 2024-03-04T16:26:17.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004cf435be36191873-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.519 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.519 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.519 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.519 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.519 DEBUG:boto:Method: GET 2024-03-04T16:26:17.519 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.520 DEBUG:boto:Data: 2024-03-04T16:26:17.520 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.520 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.520 DEBUG:boto:Params: {} 2024-03-04T16:26:17.520 DEBUG:boto:Token: None 2024-03-04T16:26:17.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000130a1c71ad7aa27c-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.523 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.523 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.523 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.523 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.523 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.523 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.523 DEBUG:boto:Data: 2024-03-04T16:26:17.523 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.523 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.523 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.524 DEBUG:boto:Params: {} 2024-03-04T16:26:17.524 DEBUG:boto:Token: None 2024-03-04T16:26:17.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NeXsY0eVBeKEtChTeFHkIDygO2I= 2024-03-04T16:26:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NeXsY0eVBeKEtChTeFHkIDygO2I=', 'Content-Length': '0'} 2024-03-04T16:26:17.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ac445e42c828bd8-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.526 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.526 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.526 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.526 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.526 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.526 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.526 DEBUG:boto:Data: 2024-03-04T16:26:17.527 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.527 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.527 DEBUG:boto:Params: {} 2024-03-04T16:26:17.527 DEBUG:boto:Token: None 2024-03-04T16:26:17.527 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NeXsY0eVBeKEtChTeFHkIDygO2I= 2024-03-04T16:26:17.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NeXsY0eVBeKEtChTeFHkIDygO2I=', 'Content-Length': '0'} 2024-03-04T16:26:17.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000731693b2d21d0909-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:17.529 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.530 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.530 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.530 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.530 DEBUG:boto:Method: GET 2024-03-04T16:26:17.530 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.530 DEBUG:boto:Data: 2024-03-04T16:26:17.530 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.530 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.530 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.530 DEBUG:boto:Params: {} 2024-03-04T16:26:17.530 DEBUG:boto:Token: None 2024-03-04T16:26:17.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb490ed1a5b6de10-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.533 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.533 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.533 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.533 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.533 DEBUG:boto:Method: GET 2024-03-04T16:26:17.533 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.533 DEBUG:boto:Data: 2024-03-04T16:26:17.533 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.533 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.533 DEBUG:boto:Params: {} 2024-03-04T16:26:17.534 DEBUG:boto:Token: None 2024-03-04T16:26:17.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c53c7e08f8255ef-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.536 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.536 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.536 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.536 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.537 DEBUG:boto:Method: GET 2024-03-04T16:26:17.537 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.537 DEBUG:boto:Data: 2024-03-04T16:26:17.537 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.537 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.537 DEBUG:boto:Params: {} 2024-03-04T16:26:17.537 DEBUG:boto:Token: None 2024-03-04T16:26:17.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.537 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c25eff62cc2efb0-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.540 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.540 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.540 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.540 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.541 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.541 DEBUG:boto:Method: GET 2024-03-04T16:26:17.541 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.541 DEBUG:boto:Data: 2024-03-04T16:26:17.541 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.541 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.541 DEBUG:boto:Params: {} 2024-03-04T16:26:17.541 DEBUG:boto:Token: None 2024-03-04T16:26:17.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.541 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b47d02ca24b9ed4-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.588 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.589 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.589 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.589 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.589 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:17.589 DEBUG:boto:Data: 2024-03-04T16:26:17.589 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.589 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.589 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.589 DEBUG:boto:Params: {} 2024-03-04T16:26:17.589 DEBUG:boto:Token: None 2024-03-04T16:26:17.590 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy 2024-03-04T16:26:17.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QdTnPvlaGABU893T5dp+uWQeSVc= 2024-03-04T16:26:17.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QdTnPvlaGABU893T5dp+uWQeSVc=', 'Content-Length': '0'} 2024-03-04T16:26:17.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eac50ccd32865d34-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.592 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.592 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.592 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.593 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:17.593 DEBUG:boto:Data: 2024-03-04T16:26:17.593 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.593 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.593 DEBUG:boto:Params: {} 2024-03-04T16:26:17.593 DEBUG:boto:Token: None 2024-03-04T16:26:17.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy 2024-03-04T16:26:17.593 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QdTnPvlaGABU893T5dp+uWQeSVc= 2024-03-04T16:26:17.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QdTnPvlaGABU893T5dp+uWQeSVc=', 'Content-Length': '0'} 2024-03-04T16:26:17.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007f29aca7156a7d6-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:17.596 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.597 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.597 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.597 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.597 DEBUG:boto:Method: GET 2024-03-04T16:26:17.597 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.597 DEBUG:boto:Data: 2024-03-04T16:26:17.597 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.597 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.597 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.597 DEBUG:boto:Params: {} 2024-03-04T16:26:17.597 DEBUG:boto:Token: None 2024-03-04T16:26:17.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb7cd94ef54e0185-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.600 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:17.600 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:17.600 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.600 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.600 DEBUG:boto:Method: GET 2024-03-04T16:26:17.600 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.600 DEBUG:boto:Data: 2024-03-04T16:26:17.601 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.601 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.601 DEBUG:boto:Params: {} 2024-03-04T16:26:17.601 DEBUG:boto:Token: None 2024-03-04T16:26:17.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:17.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A= 2024-03-04T16:26:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OtmMn0KNoguJmoKxbvjgb9ujr0A=', 'Content-Length': '0'} 2024-03-04T16:26:17.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044d7b49947c5cc6f-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.603 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.604 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.604 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.604 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.604 DEBUG:boto:Method: GET 2024-03-04T16:26:17.604 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.604 DEBUG:boto:Data: 2024-03-04T16:26:17.604 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.604 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.604 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.604 DEBUG:boto:Params: {} 2024-03-04T16:26:17.605 DEBUG:boto:Token: None 2024-03-04T16:26:17.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cb164f27212f7c-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.607 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.607 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.607 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.608 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.608 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.608 DEBUG:boto:Method: GET 2024-03-04T16:26:17.608 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:17.608 DEBUG:boto:Data: 2024-03-04T16:26:17.608 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.608 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.608 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.608 DEBUG:boto:Params: {} 2024-03-04T16:26:17.609 DEBUG:boto:Token: None 2024-03-04T16:26:17.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.609 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb4dcb0ef38b902-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.612 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:17.615 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.615 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.615 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.615 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.615 DEBUG:boto:Method: GET 2024-03-04T16:26:17.615 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.615 DEBUG:boto:Data: 2024-03-04T16:26:17.616 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.616 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.616 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.616 DEBUG:boto:Params: {} 2024-03-04T16:26:17.616 DEBUG:boto:Token: None 2024-03-04T16:26:17.616 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.616 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7a67730251ebd07-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.618 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.618 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.618 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.619 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.619 DEBUG:boto:Method: GET 2024-03-04T16:26:17.619 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.619 DEBUG:boto:Data: 2024-03-04T16:26:17.619 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.619 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.619 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.619 DEBUG:boto:Params: {} 2024-03-04T16:26:17.619 DEBUG:boto:Token: None 2024-03-04T16:26:17.620 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.620 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab4e017def52a553-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.622 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.622 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.622 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.622 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.623 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.623 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.623 DEBUG:boto:Data: 2024-03-04T16:26:17.623 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.623 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.623 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.623 DEBUG:boto:Params: {} 2024-03-04T16:26:17.623 DEBUG:boto:Token: None 2024-03-04T16:26:17.623 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.624 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZgwVnMDA/wfnD0p+NgJLusJq4do= 2024-03-04T16:26:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZgwVnMDA/wfnD0p+NgJLusJq4do=', 'Content-Length': '0'} 2024-03-04T16:26:17.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4817c114fed45d7-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.625 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.626 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.626 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.626 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.626 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.626 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.626 DEBUG:boto:Data: 2024-03-04T16:26:17.626 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.626 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.626 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.626 DEBUG:boto:Params: {} 2024-03-04T16:26:17.626 DEBUG:boto:Token: None 2024-03-04T16:26:17.627 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.627 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZgwVnMDA/wfnD0p+NgJLusJq4do= 2024-03-04T16:26:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZgwVnMDA/wfnD0p+NgJLusJq4do=', 'Content-Length': '0'} 2024-03-04T16:26:17.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aa8e8c683f336a6-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:17.629 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.629 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.629 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.629 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.629 DEBUG:boto:Method: GET 2024-03-04T16:26:17.629 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.629 DEBUG:boto:Data: 2024-03-04T16:26:17.629 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.630 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.630 DEBUG:boto:Params: {} 2024-03-04T16:26:17.630 DEBUG:boto:Token: None 2024-03-04T16:26:17.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020fdba62dfb8ed1b-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.672 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.672 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.672 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.673 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.673 DEBUG:boto:Method: GET 2024-03-04T16:26:17.673 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.673 DEBUG:boto:Data: 2024-03-04T16:26:17.673 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.673 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.673 DEBUG:boto:Params: {} 2024-03-04T16:26:17.673 DEBUG:boto:Token: None 2024-03-04T16:26:17.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006567c8896ba00481-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.676 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.676 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.677 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.677 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.677 DEBUG:boto:Method: GET 2024-03-04T16:26:17.677 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.677 DEBUG:boto:Data: 2024-03-04T16:26:17.677 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.677 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.677 DEBUG:boto:Params: {} 2024-03-04T16:26:17.677 DEBUG:boto:Token: None 2024-03-04T16:26:17.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007502de32ca8a1402-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.724 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.725 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.725 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.725 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.725 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.725 DEBUG:boto:Method: GET 2024-03-04T16:26:17.725 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.725 DEBUG:boto:Data: 2024-03-04T16:26:17.725 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.725 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.725 DEBUG:boto:Params: {} 2024-03-04T16:26:17.726 DEBUG:boto:Token: None 2024-03-04T16:26:17.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e1b467e88f672c7-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.729 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.729 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.729 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.729 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.729 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.729 DEBUG:boto:Data: 2024-03-04T16:26:17.729 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.729 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.729 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.730 DEBUG:boto:Params: {} 2024-03-04T16:26:17.730 DEBUG:boto:Token: None 2024-03-04T16:26:17.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6CtYTy2cpjLRhV0JCZPoZ2oatfs= 2024-03-04T16:26:17.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6CtYTy2cpjLRhV0JCZPoZ2oatfs=', 'Content-Length': '0'} 2024-03-04T16:26:17.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056e3a7349c5f9265-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.732 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.732 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.732 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.733 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.733 DEBUG:boto:Data: 2024-03-04T16:26:17.733 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.733 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.733 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.733 DEBUG:boto:Params: {} 2024-03-04T16:26:17.733 DEBUG:boto:Token: None 2024-03-04T16:26:17.733 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.733 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6CtYTy2cpjLRhV0JCZPoZ2oatfs= 2024-03-04T16:26:17.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6CtYTy2cpjLRhV0JCZPoZ2oatfs=', 'Content-Length': '0'} 2024-03-04T16:26:17.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000470418e69fee7591-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:17.736 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.736 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.736 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.736 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.737 DEBUG:boto:Method: GET 2024-03-04T16:26:17.737 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.737 DEBUG:boto:Data: 2024-03-04T16:26:17.737 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.737 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.737 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.737 DEBUG:boto:Params: {} 2024-03-04T16:26:17.737 DEBUG:boto:Token: None 2024-03-04T16:26:17.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd552734cdb14a4f-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.780 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.780 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:17.781 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.781 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.781 DEBUG:boto:Method: GET 2024-03-04T16:26:17.781 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.781 DEBUG:boto:Data: 2024-03-04T16:26:17.781 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.781 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.781 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.782 DEBUG:boto:Params: {} 2024-03-04T16:26:17.782 DEBUG:boto:Token: None 2024-03-04T16:26:17.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:17.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU= 2024-03-04T16:26:17.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZS9l7a8xHyLiqsV/geW8jZIy1YU=', 'Content-Length': '0'} 2024-03-04T16:26:17.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afb4adf7bf42b175-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.785 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.785 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.785 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.785 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.786 DEBUG:boto:Method: GET 2024-03-04T16:26:17.786 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.786 DEBUG:boto:Data: 2024-03-04T16:26:17.786 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.786 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.786 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.786 DEBUG:boto:Params: {} 2024-03-04T16:26:17.787 DEBUG:boto:Token: None 2024-03-04T16:26:17.787 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.787 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009935689d1dfb3eb9-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.836 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.837 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.837 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.837 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.837 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.837 DEBUG:boto:Method: GET 2024-03-04T16:26:17.837 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:17.837 DEBUG:boto:Data: 2024-03-04T16:26:17.837 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.838 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.838 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.838 DEBUG:boto:Params: {} 2024-03-04T16:26:17.838 DEBUG:boto:Token: None 2024-03-04T16:26:17.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e05abc234bc6b9-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.841 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:17.842 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.842 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.842 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.842 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.842 DEBUG:boto:Method: GET 2024-03-04T16:26:17.842 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.842 DEBUG:boto:Data: 2024-03-04T16:26:17.842 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.843 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.843 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.843 DEBUG:boto:Params: {} 2024-03-04T16:26:17.843 DEBUG:boto:Token: None 2024-03-04T16:26:17.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.843 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY= 2024-03-04T16:26:17.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY=', 'Content-Length': '0'} 2024-03-04T16:26:17.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060503732e3707721-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.845 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.846 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.846 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.846 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.846 DEBUG:boto:Method: GET 2024-03-04T16:26:17.846 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.846 DEBUG:boto:Data: 2024-03-04T16:26:17.846 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.846 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.846 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.846 DEBUG:boto:Params: {} 2024-03-04T16:26:17.846 DEBUG:boto:Token: None 2024-03-04T16:26:17.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY= 2024-03-04T16:26:17.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY=', 'Content-Length': '0'} 2024-03-04T16:26:17.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da1067fe6df8bb3b-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.849 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.849 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.849 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.849 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.850 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.850 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.850 DEBUG:boto:Data: 2024-03-04T16:26:17.850 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.850 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.850 DEBUG:boto:Params: {} 2024-03-04T16:26:17.850 DEBUG:boto:Token: None 2024-03-04T16:26:17.850 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+VZWZy0wysq83LfT0p23jftEAB4= 2024-03-04T16:26:17.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+VZWZy0wysq83LfT0p23jftEAB4=', 'Content-Length': '0'} 2024-03-04T16:26:17.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d5c5c149fd377f7-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.853 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.853 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.853 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.853 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.853 DEBUG:boto:Method: HEAD 2024-03-04T16:26:17.853 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.853 DEBUG:boto:Data: 2024-03-04T16:26:17.854 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.854 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.854 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.854 DEBUG:boto:Params: {} 2024-03-04T16:26:17.854 DEBUG:boto:Token: None 2024-03-04T16:26:17.854 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.854 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+VZWZy0wysq83LfT0p23jftEAB4= 2024-03-04T16:26:17.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+VZWZy0wysq83LfT0p23jftEAB4=', 'Content-Length': '0'} 2024-03-04T16:26:17.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ab966faf94c8f64-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:17.856 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.857 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.857 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.857 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.857 DEBUG:boto:Method: GET 2024-03-04T16:26:17.857 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.857 DEBUG:boto:Data: 2024-03-04T16:26:17.857 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.858 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.858 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.858 DEBUG:boto:Params: {} 2024-03-04T16:26:17.858 DEBUG:boto:Token: None 2024-03-04T16:26:17.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.858 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY= 2024-03-04T16:26:17.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY=', 'Content-Length': '0'} 2024-03-04T16:26:17.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d39a40bc99ba9586-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.904 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.904 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:17.904 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.904 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.905 DEBUG:boto:Method: GET 2024-03-04T16:26:17.905 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.905 DEBUG:boto:Data: 2024-03-04T16:26:17.905 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.905 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.905 DEBUG:boto:Params: {} 2024-03-04T16:26:17.905 DEBUG:boto:Token: None 2024-03-04T16:26:17.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:17.906 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY= 2024-03-04T16:26:17.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uo+js1o8d6Dth4mB9dpM1ukdhuY=', 'Content-Length': '0'} 2024-03-04T16:26:17.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f54efe7e5516270-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.948 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.948 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.948 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.948 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.949 DEBUG:boto:Method: GET 2024-03-04T16:26:17.949 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.949 DEBUG:boto:Data: 2024-03-04T16:26:17.949 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.949 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:17.949 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.949 DEBUG:boto:Params: {} 2024-03-04T16:26:17.949 DEBUG:boto:Token: None 2024-03-04T16:26:17.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.950 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad96891035ddde2-0065e5f629-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:17.952 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:17.953 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:17.953 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:17.953 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.953 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.953 DEBUG:boto:Method: GET 2024-03-04T16:26:17.954 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:17.954 DEBUG:boto:Data: 2024-03-04T16:26:17.954 DEBUG:boto:Headers: {} 2024-03-04T16:26:17.954 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:17.954 DEBUG:boto:Port: 8000 2024-03-04T16:26:17.954 DEBUG:boto:Params: {} 2024-03-04T16:26:17.955 DEBUG:boto:Token: None 2024-03-04T16:26:17.955 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:17 GMT /sbnvju-19/ 2024-03-04T16:26:17.955 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8= 2024-03-04T16:26:17.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SGMSiIoAKV9GXTBxTWfHrGi+6V8=', 'Content-Length': '0'} 2024-03-04T16:26:17.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98186f3e783b098-0065e5f629-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.000 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.001 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.001 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.001 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.001 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.001 DEBUG:boto:Data: 2024-03-04T16:26:18.001 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.001 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.001 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.001 DEBUG:boto:Params: {} 2024-03-04T16:26:18.001 DEBUG:boto:Token: None 2024-03-04T16:26:18.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs= 2024-03-04T16:26:18.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs=', 'Content-Length': '0'} 2024-03-04T16:26:18.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d27b1d0bd79c628-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.004 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.004 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.004 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.005 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.005 DEBUG:boto:Data: 2024-03-04T16:26:18.005 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.005 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.005 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.005 DEBUG:boto:Params: {} 2024-03-04T16:26:18.005 DEBUG:boto:Token: None 2024-03-04T16:26:18.005 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.005 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs= 2024-03-04T16:26:18.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs=', 'Content-Length': '0'} 2024-03-04T16:26:18.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000618b7f2c946834a9-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:18.008 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.008 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.008 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.009 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.009 DEBUG:boto:Method: GET 2024-03-04T16:26:18.009 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.009 DEBUG:boto:Data: 2024-03-04T16:26:18.009 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.009 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.009 DEBUG:boto:Params: {} 2024-03-04T16:26:18.009 DEBUG:boto:Token: None 2024-03-04T16:26:18.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae3a9719d844bb6b-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.012 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.012 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.012 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.012 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.012 DEBUG:boto:Method: GET 2024-03-04T16:26:18.012 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.013 DEBUG:boto:Data: 2024-03-04T16:26:18.013 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.013 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.013 DEBUG:boto:Params: {} 2024-03-04T16:26:18.013 DEBUG:boto:Token: None 2024-03-04T16:26:18.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c282e6b5b5e8e8f8-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.016 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.016 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.016 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.016 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.016 DEBUG:boto:Method: GET 2024-03-04T16:26:18.016 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.016 DEBUG:boto:Data: 2024-03-04T16:26:18.016 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.017 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.017 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.017 DEBUG:boto:Params: {} 2024-03-04T16:26:18.017 DEBUG:boto:Token: None 2024-03-04T16:26:18.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf342e9d99288c4-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.019 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.020 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.020 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.020 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.020 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.020 DEBUG:boto:Method: GET 2024-03-04T16:26:18.020 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.021 DEBUG:boto:Data: 2024-03-04T16:26:18.021 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.021 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.021 DEBUG:boto:Params: {} 2024-03-04T16:26:18.021 DEBUG:boto:Token: None 2024-03-04T16:26:18.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.021 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4512a9f8849acef-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.024 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.024 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.025 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.025 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.025 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.025 DEBUG:boto:Method: GET 2024-03-04T16:26:18.025 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.025 DEBUG:boto:Data: 2024-03-04T16:26:18.025 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.025 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.025 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.025 DEBUG:boto:Params: {} 2024-03-04T16:26:18.027 DEBUG:boto:Token: None 2024-03-04T16:26:18.027 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.027 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f770b12433ebf3-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.072 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.072 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.072 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.072 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.073 DEBUG:boto:Method: GET 2024-03-04T16:26:18.073 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.073 DEBUG:boto:Data: 2024-03-04T16:26:18.073 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.073 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.073 DEBUG:boto:Params: {} 2024-03-04T16:26:18.073 DEBUG:boto:Token: None 2024-03-04T16:26:18.073 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.073 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000215dfc78bd5d49da-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.076 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.076 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.076 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.076 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.076 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.076 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.077 DEBUG:boto:Data: 2024-03-04T16:26:18.077 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.077 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.077 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.077 DEBUG:boto:Params: {} 2024-03-04T16:26:18.077 DEBUG:boto:Token: None 2024-03-04T16:26:18.077 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw= 2024-03-04T16:26:18.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw=', 'Content-Length': '0'} 2024-03-04T16:26:18.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf3f3489a02a9f40-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.079 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.079 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.079 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.079 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.079 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.079 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.079 DEBUG:boto:Data: 2024-03-04T16:26:18.080 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.080 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.080 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.080 DEBUG:boto:Params: {} 2024-03-04T16:26:18.080 DEBUG:boto:Token: None 2024-03-04T16:26:18.080 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw= 2024-03-04T16:26:18.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw=', 'Content-Length': '0'} 2024-03-04T16:26:18.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecb261c5b39c1883-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.082 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.082 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.082 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.083 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.083 DEBUG:boto:Method: GET 2024-03-04T16:26:18.083 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.083 DEBUG:boto:Data: 2024-03-04T16:26:18.083 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.083 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.083 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.083 DEBUG:boto:Params: {} 2024-03-04T16:26:18.083 DEBUG:boto:Token: None 2024-03-04T16:26:18.083 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.084 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020f112afe23fda50-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.128 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.128 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.128 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.128 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.128 DEBUG:boto:Method: GET 2024-03-04T16:26:18.128 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.128 DEBUG:boto:Data: 2024-03-04T16:26:18.129 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.129 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.129 DEBUG:boto:Params: {} 2024-03-04T16:26:18.129 DEBUG:boto:Token: None 2024-03-04T16:26:18.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c19df4b9cbf30ea-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.131 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.131 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.131 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.132 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.132 DEBUG:boto:Method: GET 2024-03-04T16:26:18.132 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.132 DEBUG:boto:Data: 2024-03-04T16:26:18.132 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.132 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.132 DEBUG:boto:Params: {} 2024-03-04T16:26:18.132 DEBUG:boto:Token: None 2024-03-04T16:26:18.132 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa19a08048c611e-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.176 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.176 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.177 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.177 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.177 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.177 DEBUG:boto:Method: GET 2024-03-04T16:26:18.177 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.177 DEBUG:boto:Data: 2024-03-04T16:26:18.177 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.177 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.177 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.177 DEBUG:boto:Params: {} 2024-03-04T16:26:18.177 DEBUG:boto:Token: None 2024-03-04T16:26:18.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b7afaa45fe654e-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.180 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.181 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.181 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.181 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.181 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.181 DEBUG:boto:Data: 2024-03-04T16:26:18.181 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.181 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.181 DEBUG:boto:Params: {} 2024-03-04T16:26:18.181 DEBUG:boto:Token: None 2024-03-04T16:26:18.181 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU= 2024-03-04T16:26:18.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU=', 'Content-Length': '0'} 2024-03-04T16:26:18.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2d0cc1de8d1d3d0-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.184 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.184 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.184 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.184 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.184 DEBUG:boto:Data: 2024-03-04T16:26:18.184 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.184 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.184 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.184 DEBUG:boto:Params: {} 2024-03-04T16:26:18.184 DEBUG:boto:Token: None 2024-03-04T16:26:18.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.185 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU= 2024-03-04T16:26:18.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU=', 'Content-Length': '0'} 2024-03-04T16:26:18.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000682330431bdfef6d-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.187 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.187 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.188 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.188 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.188 DEBUG:boto:Method: GET 2024-03-04T16:26:18.188 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.188 DEBUG:boto:Data: 2024-03-04T16:26:18.188 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.188 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.188 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.188 DEBUG:boto:Params: {} 2024-03-04T16:26:18.188 DEBUG:boto:Token: None 2024-03-04T16:26:18.188 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e71169f0b46449c-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.232 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.232 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.232 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.232 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.233 DEBUG:boto:Method: GET 2024-03-04T16:26:18.233 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.233 DEBUG:boto:Data: 2024-03-04T16:26:18.233 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.233 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.233 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.233 DEBUG:boto:Params: {} 2024-03-04T16:26:18.233 DEBUG:boto:Token: None 2024-03-04T16:26:18.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b769cbb427daeee-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.236 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.236 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.236 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.236 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.236 DEBUG:boto:Method: GET 2024-03-04T16:26:18.236 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.236 DEBUG:boto:Data: 2024-03-04T16:26:18.236 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.236 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:18.236 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.236 DEBUG:boto:Params: {} 2024-03-04T16:26:18.237 DEBUG:boto:Token: None 2024-03-04T16:26:18.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd63c8f40ee6404-0065e5f62a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.239 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.239 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.239 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.240 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.240 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.240 DEBUG:boto:Method: GET 2024-03-04T16:26:18.240 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.240 DEBUG:boto:Data: 2024-03-04T16:26:18.240 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.240 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.240 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.240 DEBUG:boto:Params: {} 2024-03-04T16:26:18.240 DEBUG:boto:Token: None 2024-03-04T16:26:18.240 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.240 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41222b2b5021494-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.243 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.244 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone2, test-zone1} 2024-03-04T16:26:18.244 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone3, test-zone1} 2024-03-04T16:26:18.244 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.244 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.244 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.244 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:18.244 DEBUG:boto:Data: 2024-03-04T16:26:18.244 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.244 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.245 DEBUG:boto:Params: {} 2024-03-04T16:26:18.245 DEBUG:boto:Token: None 2024-03-04T16:26:18.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gjKmBvPrn0QFt2Jr39tz9V7Pe6M= 2024-03-04T16:26:18.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gjKmBvPrn0QFt2Jr39tz9V7Pe6M=', 'Content-Length': '0'} 2024-03-04T16:26:18.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9e75e2b8214368a-0065e5f62a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.248 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.248 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.249 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.249 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:18.249 DEBUG:boto:Data: 2024-03-04T16:26:18.249 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.249 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.249 DEBUG:boto:Params: {} 2024-03-04T16:26:18.249 DEBUG:boto:Token: None 2024-03-04T16:26:18.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gjKmBvPrn0QFt2Jr39tz9V7Pe6M= 2024-03-04T16:26:18.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gjKmBvPrn0QFt2Jr39tz9V7Pe6M=', 'Content-Length': '0'} 2024-03-04T16:26:18.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016538974337cb1ba-0065e5f62a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:18.252 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.252 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.252 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.252 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.252 DEBUG:boto:Method: GET 2024-03-04T16:26:18.252 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.253 DEBUG:boto:Data: 2024-03-04T16:26:18.253 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.253 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.253 DEBUG:boto:Params: {} 2024-03-04T16:26:18.253 DEBUG:boto:Token: None 2024-03-04T16:26:18.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/?versions 2024-03-04T16:26:18.253 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM= 2024-03-04T16:26:18.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM=', 'Content-Length': '0'} 2024-03-04T16:26:18.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ba10704633c510-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.256 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:18.257 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:18.257 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.257 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.257 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.258 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.258 DEBUG:boto:Method: GET 2024-03-04T16:26:18.258 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.258 DEBUG:boto:Data: 2024-03-04T16:26:18.258 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.258 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.258 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.258 DEBUG:boto:Params: {} 2024-03-04T16:26:18.258 DEBUG:boto:Token: None 2024-03-04T16:26:18.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/?versions 2024-03-04T16:26:18.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM= 2024-03-04T16:26:18.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM=', 'Content-Length': '0'} 2024-03-04T16:26:18.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab55b36661ca1ab-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.262 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:18.262 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.262 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.262 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.263 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.263 DEBUG:boto:Method: GET 2024-03-04T16:26:18.263 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.263 DEBUG:boto:Data: 2024-03-04T16:26:18.263 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.263 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.263 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.263 DEBUG:boto:Params: {} 2024-03-04T16:26:18.263 DEBUG:boto:Token: None 2024-03-04T16:26:18.263 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/?versions 2024-03-04T16:26:18.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM= 2024-03-04T16:26:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM=', 'Content-Length': '0'} 2024-03-04T16:26:18.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f3141149891498-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.266 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.267 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.267 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.267 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.267 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.267 DEBUG:boto:Method: GET 2024-03-04T16:26:18.267 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:18.267 DEBUG:boto:Data: 2024-03-04T16:26:18.267 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.268 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.268 DEBUG:boto:Params: {} 2024-03-04T16:26:18.268 DEBUG:boto:Token: None 2024-03-04T16:26:18.268 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/?versions 2024-03-04T16:26:18.268 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM= 2024-03-04T16:26:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y5mPDFvRlUVf464orfde9AN1sMM=', 'Content-Length': '0'} 2024-03-04T16:26:18.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a6e480b52bf750e-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.271 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:18.272 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.272 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.272 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.272 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.272 DEBUG:boto:Method: GET 2024-03-04T16:26:18.272 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.272 DEBUG:boto:Data: 2024-03-04T16:26:18.272 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.272 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.272 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.273 DEBUG:boto:Params: {} 2024-03-04T16:26:18.273 DEBUG:boto:Token: None 2024-03-04T16:26:18.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a19db6ec6c5a4b2-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.275 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.275 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.275 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.275 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.275 DEBUG:boto:Method: GET 2024-03-04T16:26:18.275 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.276 DEBUG:boto:Data: 2024-03-04T16:26:18.276 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.276 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.276 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.276 DEBUG:boto:Params: {} 2024-03-04T16:26:18.276 DEBUG:boto:Token: None 2024-03-04T16:26:18.276 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.276 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb37f41140155561-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.279 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.279 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.279 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.279 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.279 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.279 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.279 DEBUG:boto:Data: 2024-03-04T16:26:18.279 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.279 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.279 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.279 DEBUG:boto:Params: {} 2024-03-04T16:26:18.280 DEBUG:boto:Token: None 2024-03-04T16:26:18.280 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nmzwtqaOPAlF1sH8gN17jf+Is34= 2024-03-04T16:26:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nmzwtqaOPAlF1sH8gN17jf+Is34=', 'Content-Length': '0'} 2024-03-04T16:26:18.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000645b6cddcb247fb5-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.282 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.282 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.282 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.282 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.282 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.282 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.282 DEBUG:boto:Data: 2024-03-04T16:26:18.282 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.282 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.282 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.283 DEBUG:boto:Params: {} 2024-03-04T16:26:18.283 DEBUG:boto:Token: None 2024-03-04T16:26:18.283 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.283 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nmzwtqaOPAlF1sH8gN17jf+Is34= 2024-03-04T16:26:18.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nmzwtqaOPAlF1sH8gN17jf+Is34=', 'Content-Length': '0'} 2024-03-04T16:26:18.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b22933909572425-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:18.285 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.285 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.285 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.285 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.285 DEBUG:boto:Method: GET 2024-03-04T16:26:18.286 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.286 DEBUG:boto:Data: 2024-03-04T16:26:18.286 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.286 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.286 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.286 DEBUG:boto:Params: {} 2024-03-04T16:26:18.286 DEBUG:boto:Token: None 2024-03-04T16:26:18.286 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013c89e4c60ff3d8e-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.288 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.288 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.288 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.289 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.289 DEBUG:boto:Method: GET 2024-03-04T16:26:18.289 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.289 DEBUG:boto:Data: 2024-03-04T16:26:18.289 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.289 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.289 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.289 DEBUG:boto:Params: {} 2024-03-04T16:26:18.289 DEBUG:boto:Token: None 2024-03-04T16:26:18.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e700c8202ca5f855-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.292 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.292 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.292 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.292 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.292 DEBUG:boto:Method: GET 2024-03-04T16:26:18.292 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.292 DEBUG:boto:Data: 2024-03-04T16:26:18.292 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.292 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.293 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.293 DEBUG:boto:Params: {} 2024-03-04T16:26:18.293 DEBUG:boto:Token: None 2024-03-04T16:26:18.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.293 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbd74318f391fe17-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.336 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.336 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.337 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.337 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.337 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.337 DEBUG:boto:Method: GET 2024-03-04T16:26:18.337 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.337 DEBUG:boto:Data: 2024-03-04T16:26:18.337 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.337 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.337 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.337 DEBUG:boto:Params: {} 2024-03-04T16:26:18.338 DEBUG:boto:Token: None 2024-03-04T16:26:18.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000679fe558c696479d-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.341 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.341 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.341 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.341 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.341 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:18.341 DEBUG:boto:Data: 2024-03-04T16:26:18.341 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.342 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.342 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.342 DEBUG:boto:Params: {} 2024-03-04T16:26:18.342 DEBUG:boto:Token: None 2024-03-04T16:26:18.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy 2024-03-04T16:26:18.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j6ZgLAD4dsvBmQ4j8CIOxvnwkM8= 2024-03-04T16:26:18.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j6ZgLAD4dsvBmQ4j8CIOxvnwkM8=', 'Content-Length': '0'} 2024-03-04T16:26:18.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007882d225475bb332-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.344 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.344 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.344 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.345 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:18.345 DEBUG:boto:Data: 2024-03-04T16:26:18.345 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.345 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.345 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.345 DEBUG:boto:Params: {} 2024-03-04T16:26:18.345 DEBUG:boto:Token: None 2024-03-04T16:26:18.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy 2024-03-04T16:26:18.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j6ZgLAD4dsvBmQ4j8CIOxvnwkM8= 2024-03-04T16:26:18.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j6ZgLAD4dsvBmQ4j8CIOxvnwkM8=', 'Content-Length': '0'} 2024-03-04T16:26:18.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b28fd2d1b6772b0d-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:18.348 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.348 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.348 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.348 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.348 DEBUG:boto:Method: GET 2024-03-04T16:26:18.348 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.349 DEBUG:boto:Data: 2024-03-04T16:26:18.349 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.349 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.349 DEBUG:boto:Params: {} 2024-03-04T16:26:18.349 DEBUG:boto:Token: None 2024-03-04T16:26:18.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000751c84c5723131f5-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.351 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:18.351 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:18.351 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.351 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.351 DEBUG:boto:Method: GET 2024-03-04T16:26:18.352 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.352 DEBUG:boto:Data: 2024-03-04T16:26:18.352 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.352 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.352 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.352 DEBUG:boto:Params: {} 2024-03-04T16:26:18.352 DEBUG:boto:Token: None 2024-03-04T16:26:18.352 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:18.352 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs= 2024-03-04T16:26:18.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i0XPjBq2N1vFgfjyUFCIC0Z+NAs=', 'Content-Length': '0'} 2024-03-04T16:26:18.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000792de41d8300cd91-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.354 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.354 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.354 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.355 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.355 DEBUG:boto:Method: GET 2024-03-04T16:26:18.355 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.355 DEBUG:boto:Data: 2024-03-04T16:26:18.355 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.355 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.355 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.355 DEBUG:boto:Params: {} 2024-03-04T16:26:18.355 DEBUG:boto:Token: None 2024-03-04T16:26:18.355 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.356 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c18dcbe385c796-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.358 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.359 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.359 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.359 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.359 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.359 DEBUG:boto:Method: GET 2024-03-04T16:26:18.359 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:18.359 DEBUG:boto:Data: 2024-03-04T16:26:18.359 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.359 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.359 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.359 DEBUG:boto:Params: {} 2024-03-04T16:26:18.360 DEBUG:boto:Token: None 2024-03-04T16:26:18.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.360 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e1993dc4a471cc-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.362 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:18.363 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.363 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.363 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.363 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.363 DEBUG:boto:Method: GET 2024-03-04T16:26:18.363 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.363 DEBUG:boto:Data: 2024-03-04T16:26:18.364 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.364 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.364 DEBUG:boto:Params: {} 2024-03-04T16:26:18.364 DEBUG:boto:Token: None 2024-03-04T16:26:18.364 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.364 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001adbbcce35231fb-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.408 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.408 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.408 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.408 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.409 DEBUG:boto:Method: GET 2024-03-04T16:26:18.409 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.409 DEBUG:boto:Data: 2024-03-04T16:26:18.409 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.409 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.409 DEBUG:boto:Params: {} 2024-03-04T16:26:18.409 DEBUG:boto:Token: None 2024-03-04T16:26:18.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067f0714495787e28-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.412 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.412 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.412 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.412 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.412 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.412 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.412 DEBUG:boto:Data: 2024-03-04T16:26:18.412 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.413 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.413 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.413 DEBUG:boto:Params: {} 2024-03-04T16:26:18.413 DEBUG:boto:Token: None 2024-03-04T16:26:18.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lLxGrwWod3jbBOnsq9yEAPDTvUc= 2024-03-04T16:26:18.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lLxGrwWod3jbBOnsq9yEAPDTvUc=', 'Content-Length': '0'} 2024-03-04T16:26:18.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600fe711e7d86a55-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.415 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.415 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.415 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.415 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.415 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.415 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.416 DEBUG:boto:Data: 2024-03-04T16:26:18.416 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.416 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.416 DEBUG:boto:Params: {} 2024-03-04T16:26:18.416 DEBUG:boto:Token: None 2024-03-04T16:26:18.416 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.416 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lLxGrwWod3jbBOnsq9yEAPDTvUc= 2024-03-04T16:26:18.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lLxGrwWod3jbBOnsq9yEAPDTvUc=', 'Content-Length': '0'} 2024-03-04T16:26:18.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000869cee70b762d3a3-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:18.418 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.418 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.419 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.419 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.419 DEBUG:boto:Method: GET 2024-03-04T16:26:18.419 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.419 DEBUG:boto:Data: 2024-03-04T16:26:18.419 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.419 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.419 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.419 DEBUG:boto:Params: {} 2024-03-04T16:26:18.419 DEBUG:boto:Token: None 2024-03-04T16:26:18.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.420 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd6386732c94c757-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.464 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.464 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.464 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.464 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.465 DEBUG:boto:Method: GET 2024-03-04T16:26:18.465 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.465 DEBUG:boto:Data: 2024-03-04T16:26:18.465 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.465 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.465 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.465 DEBUG:boto:Params: {} 2024-03-04T16:26:18.465 DEBUG:boto:Token: None 2024-03-04T16:26:18.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d787c6d5480227c8-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.467 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.468 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.468 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.468 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.468 DEBUG:boto:Method: GET 2024-03-04T16:26:18.468 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.468 DEBUG:boto:Data: 2024-03-04T16:26:18.468 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.468 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.468 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.468 DEBUG:boto:Params: {} 2024-03-04T16:26:18.469 DEBUG:boto:Token: None 2024-03-04T16:26:18.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.469 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6c034a90ebc951-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.471 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.472 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.472 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.472 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.472 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.472 DEBUG:boto:Method: GET 2024-03-04T16:26:18.472 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.472 DEBUG:boto:Data: 2024-03-04T16:26:18.473 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.473 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.473 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.473 DEBUG:boto:Params: {} 2024-03-04T16:26:18.473 DEBUG:boto:Token: None 2024-03-04T16:26:18.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083550398f76ca77b-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.476 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.476 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.476 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.476 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.477 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.477 DEBUG:boto:Data: 2024-03-04T16:26:18.477 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.477 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.477 DEBUG:boto:Params: {} 2024-03-04T16:26:18.477 DEBUG:boto:Token: None 2024-03-04T16:26:18.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2KnGPmVCC7R9tHc4knITWdNL1Q0= 2024-03-04T16:26:18.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2KnGPmVCC7R9tHc4knITWdNL1Q0=', 'Content-Length': '0'} 2024-03-04T16:26:18.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fb826bd67109aa6-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.479 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.480 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.480 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.480 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.480 DEBUG:boto:Data: 2024-03-04T16:26:18.480 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.480 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.480 DEBUG:boto:Params: {} 2024-03-04T16:26:18.480 DEBUG:boto:Token: None 2024-03-04T16:26:18.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2KnGPmVCC7R9tHc4knITWdNL1Q0= 2024-03-04T16:26:18.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2KnGPmVCC7R9tHc4knITWdNL1Q0=', 'Content-Length': '0'} 2024-03-04T16:26:18.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b04c93de25d01a32-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:18.483 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.483 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.483 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.483 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.483 DEBUG:boto:Method: GET 2024-03-04T16:26:18.483 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.483 DEBUG:boto:Data: 2024-03-04T16:26:18.484 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.484 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.484 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.484 DEBUG:boto:Params: {} 2024-03-04T16:26:18.484 DEBUG:boto:Token: None 2024-03-04T16:26:18.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.484 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004b78984c86cd449-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.528 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.528 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:18.528 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.528 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.529 DEBUG:boto:Method: GET 2024-03-04T16:26:18.529 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.529 DEBUG:boto:Data: 2024-03-04T16:26:18.529 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.529 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.529 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.529 DEBUG:boto:Params: {} 2024-03-04T16:26:18.529 DEBUG:boto:Token: None 2024-03-04T16:26:18.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:18.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ= 2024-03-04T16:26:18.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwBzf0prML4xaB70cABvbhRZmQ=', 'Content-Length': '0'} 2024-03-04T16:26:18.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b73af71823a25e4-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.532 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.532 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.532 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.532 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.532 DEBUG:boto:Method: GET 2024-03-04T16:26:18.533 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.533 DEBUG:boto:Data: 2024-03-04T16:26:18.533 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.533 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.533 DEBUG:boto:Params: {} 2024-03-04T16:26:18.533 DEBUG:boto:Token: None 2024-03-04T16:26:18.533 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a31fc94d6b354bc4-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.536 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.537 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.537 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.537 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.537 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.537 DEBUG:boto:Method: GET 2024-03-04T16:26:18.537 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:18.537 DEBUG:boto:Data: 2024-03-04T16:26:18.537 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.537 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.538 DEBUG:boto:Params: {} 2024-03-04T16:26:18.538 DEBUG:boto:Token: None 2024-03-04T16:26:18.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701a4db3dd5f4d43-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.540 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:18.541 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.541 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.541 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.541 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.541 DEBUG:boto:Method: GET 2024-03-04T16:26:18.541 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.541 DEBUG:boto:Data: 2024-03-04T16:26:18.541 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.542 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.542 DEBUG:boto:Params: {} 2024-03-04T16:26:18.542 DEBUG:boto:Token: None 2024-03-04T16:26:18.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f9c64b94ac9f99-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.588 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.588 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.588 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.588 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.588 DEBUG:boto:Method: GET 2024-03-04T16:26:18.589 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.589 DEBUG:boto:Data: 2024-03-04T16:26:18.589 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.589 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.589 DEBUG:boto:Params: {} 2024-03-04T16:26:18.589 DEBUG:boto:Token: None 2024-03-04T16:26:18.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.589 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000862d3ad46e301419-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.591 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.592 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.592 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.592 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.592 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.592 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.592 DEBUG:boto:Data: 2024-03-04T16:26:18.592 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.592 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.592 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.592 DEBUG:boto:Params: {} 2024-03-04T16:26:18.593 DEBUG:boto:Token: None 2024-03-04T16:26:18.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.593 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rNNxg+Mbc2Cu1X6xLuvUJGFGC78= 2024-03-04T16:26:18.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rNNxg+Mbc2Cu1X6xLuvUJGFGC78=', 'Content-Length': '0'} 2024-03-04T16:26:18.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef99ef3087e8bd6c-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.595 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.595 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.595 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.595 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.595 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.595 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.595 DEBUG:boto:Data: 2024-03-04T16:26:18.595 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.596 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.596 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.596 DEBUG:boto:Params: {} 2024-03-04T16:26:18.596 DEBUG:boto:Token: None 2024-03-04T16:26:18.596 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.596 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rNNxg+Mbc2Cu1X6xLuvUJGFGC78= 2024-03-04T16:26:18.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rNNxg+Mbc2Cu1X6xLuvUJGFGC78=', 'Content-Length': '0'} 2024-03-04T16:26:18.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013d71ed5d0f6c1dc-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:18.599 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.599 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.599 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.599 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.599 DEBUG:boto:Method: GET 2024-03-04T16:26:18.599 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.599 DEBUG:boto:Data: 2024-03-04T16:26:18.599 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.599 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.599 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.599 DEBUG:boto:Params: {} 2024-03-04T16:26:18.600 DEBUG:boto:Token: None 2024-03-04T16:26:18.600 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.600 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b66a23d50b069f9-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.648 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.648 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.648 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.649 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.649 DEBUG:boto:Method: GET 2024-03-04T16:26:18.649 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.649 DEBUG:boto:Data: 2024-03-04T16:26:18.649 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.649 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.649 DEBUG:boto:Params: {} 2024-03-04T16:26:18.649 DEBUG:boto:Token: None 2024-03-04T16:26:18.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb7ac7588583bd2b-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.692 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.692 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.692 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.693 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.693 DEBUG:boto:Method: GET 2024-03-04T16:26:18.693 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.693 DEBUG:boto:Data: 2024-03-04T16:26:18.693 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.693 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.693 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.693 DEBUG:boto:Params: {} 2024-03-04T16:26:18.693 DEBUG:boto:Token: None 2024-03-04T16:26:18.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552dac1a7e46ad20-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.696 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.697 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.697 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.697 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.697 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.697 DEBUG:boto:Method: GET 2024-03-04T16:26:18.697 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.697 DEBUG:boto:Data: 2024-03-04T16:26:18.697 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.697 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.698 DEBUG:boto:Params: {} 2024-03-04T16:26:18.698 DEBUG:boto:Token: None 2024-03-04T16:26:18.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d06ea6a7df478ad9-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.701 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.701 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.701 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.701 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.701 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.701 DEBUG:boto:Data: 2024-03-04T16:26:18.702 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.702 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.702 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.702 DEBUG:boto:Params: {} 2024-03-04T16:26:18.702 DEBUG:boto:Token: None 2024-03-04T16:26:18.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.702 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs= 2024-03-04T16:26:18.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs=', 'Content-Length': '0'} 2024-03-04T16:26:18.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000405503ec19962ecd-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.704 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.704 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.705 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.705 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.705 DEBUG:boto:Data: 2024-03-04T16:26:18.705 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.705 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.705 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.705 DEBUG:boto:Params: {} 2024-03-04T16:26:18.705 DEBUG:boto:Token: None 2024-03-04T16:26:18.705 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.705 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs= 2024-03-04T16:26:18.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i1J7Een9y0Lo71GiNqtHld+TdMs=', 'Content-Length': '0'} 2024-03-04T16:26:18.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd40371b50c673e2-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:18.708 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.708 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.708 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.708 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.709 DEBUG:boto:Method: GET 2024-03-04T16:26:18.709 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.709 DEBUG:boto:Data: 2024-03-04T16:26:18.709 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.709 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.709 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.709 DEBUG:boto:Params: {} 2024-03-04T16:26:18.709 DEBUG:boto:Token: None 2024-03-04T16:26:18.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.709 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050c4163cb010bc6d-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.756 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.756 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:18.756 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.757 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.757 DEBUG:boto:Method: GET 2024-03-04T16:26:18.757 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.757 DEBUG:boto:Data: 2024-03-04T16:26:18.757 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.757 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.757 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.757 DEBUG:boto:Params: {} 2024-03-04T16:26:18.757 DEBUG:boto:Token: None 2024-03-04T16:26:18.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:18.758 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw= 2024-03-04T16:26:18.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fjwwZtjq3fb4L4IUq3xx+3fcSpw=', 'Content-Length': '0'} 2024-03-04T16:26:18.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee15f08a0c71ec3d-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.800 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.800 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.800 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.800 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.801 DEBUG:boto:Method: GET 2024-03-04T16:26:18.801 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.801 DEBUG:boto:Data: 2024-03-04T16:26:18.801 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.801 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.801 DEBUG:boto:Params: {} 2024-03-04T16:26:18.801 DEBUG:boto:Token: None 2024-03-04T16:26:18.801 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccfdcebf22361e4-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.804 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.805 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.805 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.805 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.805 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.805 DEBUG:boto:Method: GET 2024-03-04T16:26:18.805 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:18.805 DEBUG:boto:Data: 2024-03-04T16:26:18.805 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.805 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.805 DEBUG:boto:Params: {} 2024-03-04T16:26:18.806 DEBUG:boto:Token: None 2024-03-04T16:26:18.806 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.806 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712ffde6c6ce561e-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.809 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.809 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.809 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.809 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.809 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.810 DEBUG:boto:Method: GET 2024-03-04T16:26:18.810 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.810 DEBUG:boto:Data: 2024-03-04T16:26:18.810 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.810 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.810 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.810 DEBUG:boto:Params: {} 2024-03-04T16:26:18.810 DEBUG:boto:Token: None 2024-03-04T16:26:18.810 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.810 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043652b90408a9359-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.813 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.813 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.813 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.813 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.813 DEBUG:boto:Method: GET 2024-03-04T16:26:18.813 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.813 DEBUG:boto:Data: 2024-03-04T16:26:18.813 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.813 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.813 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.813 DEBUG:boto:Params: {} 2024-03-04T16:26:18.814 DEBUG:boto:Token: None 2024-03-04T16:26:18.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cace286a1e49d07-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.816 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.816 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.816 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.816 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.816 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.817 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.817 DEBUG:boto:Data: 2024-03-04T16:26:18.817 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.817 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.817 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.817 DEBUG:boto:Params: {} 2024-03-04T16:26:18.817 DEBUG:boto:Token: None 2024-03-04T16:26:18.817 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.817 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw= 2024-03-04T16:26:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw=', 'Content-Length': '0'} 2024-03-04T16:26:18.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000317a6725f02af908-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.819 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.819 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.820 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.820 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.820 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.820 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.820 DEBUG:boto:Data: 2024-03-04T16:26:18.820 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.820 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.820 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.820 DEBUG:boto:Params: {} 2024-03-04T16:26:18.820 DEBUG:boto:Token: None 2024-03-04T16:26:18.821 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.821 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw= 2024-03-04T16:26:18.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aVqmr1Mi4AQu1F4i87g7avvZsrw=', 'Content-Length': '0'} 2024-03-04T16:26:18.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069d65782c6bec806-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.823 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.823 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.823 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.823 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.823 DEBUG:boto:Method: GET 2024-03-04T16:26:18.823 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.823 DEBUG:boto:Data: 2024-03-04T16:26:18.823 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.823 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.823 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.824 DEBUG:boto:Params: {} 2024-03-04T16:26:18.824 DEBUG:boto:Token: None 2024-03-04T16:26:18.824 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.824 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7f25a7070592403-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.826 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.826 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.826 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.826 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.826 DEBUG:boto:Method: GET 2024-03-04T16:26:18.826 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.826 DEBUG:boto:Data: 2024-03-04T16:26:18.827 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.827 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.827 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.827 DEBUG:boto:Params: {} 2024-03-04T16:26:18.827 DEBUG:boto:Token: None 2024-03-04T16:26:18.827 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.827 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009917d7c5f45119cf-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.872 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.872 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.872 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.872 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.873 DEBUG:boto:Method: GET 2024-03-04T16:26:18.873 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.873 DEBUG:boto:Data: 2024-03-04T16:26:18.873 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.873 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.873 DEBUG:boto:Params: {} 2024-03-04T16:26:18.873 DEBUG:boto:Token: None 2024-03-04T16:26:18.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06bc6ce05e699e1-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.877 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.877 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.877 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.877 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.878 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.878 DEBUG:boto:Method: GET 2024-03-04T16:26:18.878 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.878 DEBUG:boto:Data: 2024-03-04T16:26:18.878 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.878 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.878 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.878 DEBUG:boto:Params: {} 2024-03-04T16:26:18.878 DEBUG:boto:Token: None 2024-03-04T16:26:18.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf46ef3497128cf-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.881 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.882 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.882 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.882 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.882 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.882 DEBUG:boto:Data: 2024-03-04T16:26:18.882 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.882 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.882 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.882 DEBUG:boto:Params: {} 2024-03-04T16:26:18.882 DEBUG:boto:Token: None 2024-03-04T16:26:18.883 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.883 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU= 2024-03-04T16:26:18.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU=', 'Content-Length': '0'} 2024-03-04T16:26:18.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c05bec12a4630aa5-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.885 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.885 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.885 DEBUG:boto:Method: HEAD 2024-03-04T16:26:18.885 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.885 DEBUG:boto:Data: 2024-03-04T16:26:18.886 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.886 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.886 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.886 DEBUG:boto:Params: {} 2024-03-04T16:26:18.886 DEBUG:boto:Token: None 2024-03-04T16:26:18.886 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU= 2024-03-04T16:26:18.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FSzc7RDQjdCsm/jf5ybsMSvqTTU=', 'Content-Length': '0'} 2024-03-04T16:26:18.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c524c47acd4fea4-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:18.891 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.892 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.892 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.892 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.892 DEBUG:boto:Method: GET 2024-03-04T16:26:18.892 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.893 DEBUG:boto:Data: 2024-03-04T16:26:18.893 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.893 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.893 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.893 DEBUG:boto:Params: {} 2024-03-04T16:26:18.894 DEBUG:boto:Token: None 2024-03-04T16:26:18.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.894 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51cd9ad3dbdcdae-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.944 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.944 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:18.944 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.944 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.945 DEBUG:boto:Method: GET 2024-03-04T16:26:18.945 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.945 DEBUG:boto:Data: 2024-03-04T16:26:18.945 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.945 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.945 DEBUG:boto:Params: {} 2024-03-04T16:26:18.945 DEBUG:boto:Token: None 2024-03-04T16:26:18.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:18.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0= 2024-03-04T16:26:18.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AAVUAFPrglDHQSDy7tNSTdLa8z0=', 'Content-Length': '0'} 2024-03-04T16:26:18.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a115db2ebf7cd95c-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.992 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.992 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.993 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.993 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.993 DEBUG:boto:Method: GET 2024-03-04T16:26:18.993 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.993 DEBUG:boto:Data: 2024-03-04T16:26:18.993 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:18.993 DEBUG:boto:Port: 8001 2024-03-04T16:26:18.993 DEBUG:boto:Params: {} 2024-03-04T16:26:18.994 DEBUG:boto:Token: None 2024-03-04T16:26:18.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:18.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aacc7cd60300543e-0065e5f62a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:18.997 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:18.997 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:18.997 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:18.998 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.998 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.998 DEBUG:boto:Method: GET 2024-03-04T16:26:18.998 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:18.998 DEBUG:boto:Data: 2024-03-04T16:26:18.998 DEBUG:boto:Headers: {} 2024-03-04T16:26:18.998 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:18.998 DEBUG:boto:Port: 8000 2024-03-04T16:26:18.998 DEBUG:boto:Params: {} 2024-03-04T16:26:18.998 DEBUG:boto:Token: None 2024-03-04T16:26:18.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:18 GMT /sbnvju-19/ 2024-03-04T16:26:18.999 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs= 2024-03-04T16:26:18.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WI3JocZLU22nk+3dUlIw4S1eFKs=', 'Content-Length': '0'} 2024-03-04T16:26:19.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04e10d9c933e395-0065e5f62a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.001 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone3, test-zone1} 2024-03-04T16:26:19.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone3, test-zone2} 2024-03-04T16:26:19.002 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.002 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.002 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.002 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:19.003 DEBUG:boto:Data: 2024-03-04T16:26:19.003 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.003 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.003 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.003 DEBUG:boto:Params: {} 2024-03-04T16:26:19.003 DEBUG:boto:Token: None 2024-03-04T16:26:19.003 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.003 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MN34Psd6VPCVi8g+ebF9ZvF+u+w= 2024-03-04T16:26:19.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MN34Psd6VPCVi8g+ebF9ZvF+u+w=', 'Content-Length': '0'} 2024-03-04T16:26:19.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a8d15583eb47fe8-0065e5f62b-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.006 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.006 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.006 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.006 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:26:19.007 DEBUG:boto:Data: 2024-03-04T16:26:19.007 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.007 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.007 DEBUG:boto:Params: {} 2024-03-04T16:26:19.007 DEBUG:boto:Token: None 2024-03-04T16:26:19.007 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.007 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MN34Psd6VPCVi8g+ebF9ZvF+u+w= 2024-03-04T16:26:19.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MN34Psd6VPCVi8g+ebF9ZvF+u+w=', 'Content-Length': '0'} 2024-03-04T16:26:19.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e318387809e975c3-0065e5f62b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.010 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:19.010 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.010 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.010 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.011 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.011 DEBUG:boto:Method: GET 2024-03-04T16:26:19.011 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.011 DEBUG:boto:Data: 2024-03-04T16:26:19.011 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.011 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.011 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.011 DEBUG:boto:Params: {} 2024-03-04T16:26:19.011 DEBUG:boto:Token: None 2024-03-04T16:26:19.011 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/?versions 2024-03-04T16:26:19.012 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY= 2024-03-04T16:26:19.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY=', 'Content-Length': '0'} 2024-03-04T16:26:19.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007df72da065ce1b58-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.014 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:19.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:19.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:19.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:19.015 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:19.015 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.016 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.016 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.016 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.016 DEBUG:boto:Method: GET 2024-03-04T16:26:19.016 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.016 DEBUG:boto:Data: 2024-03-04T16:26:19.016 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.016 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.016 DEBUG:boto:Params: {} 2024-03-04T16:26:19.016 DEBUG:boto:Token: None 2024-03-04T16:26:19.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/?versions 2024-03-04T16:26:19.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY= 2024-03-04T16:26:19.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY=', 'Content-Length': '0'} 2024-03-04T16:26:19.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009679a0b83c6944b9-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.020 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.020 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:19.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:19.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:19.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:19.021 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.021 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.021 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.021 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.021 DEBUG:boto:Method: GET 2024-03-04T16:26:19.021 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.021 DEBUG:boto:Data: 2024-03-04T16:26:19.021 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.021 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.022 DEBUG:boto:Params: {} 2024-03-04T16:26:19.022 DEBUG:boto:Token: None 2024-03-04T16:26:19.022 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/?versions 2024-03-04T16:26:19.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY= 2024-03-04T16:26:19.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY=', 'Content-Length': '0'} 2024-03-04T16:26:19.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001609a8d550306678-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.024 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.025 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.025 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.025 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.025 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.025 DEBUG:boto:Method: GET 2024-03-04T16:26:19.025 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:26:19.025 DEBUG:boto:Data: 2024-03-04T16:26:19.026 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.026 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.026 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.026 DEBUG:boto:Params: {} 2024-03-04T16:26:19.026 DEBUG:boto:Token: None 2024-03-04T16:26:19.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/?versions 2024-03-04T16:26:19.026 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY= 2024-03-04T16:26:19.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AtyJTHbV699qhUrk1vxAeyrLWyY=', 'Content-Length': '0'} 2024-03-04T16:26:19.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca2c87413a2f4cb-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.029 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:19.030 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.030 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.030 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.030 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.030 DEBUG:boto:Method: GET 2024-03-04T16:26:19.030 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.030 DEBUG:boto:Data: 2024-03-04T16:26:19.030 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.030 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.030 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.031 DEBUG:boto:Params: {} 2024-03-04T16:26:19.031 DEBUG:boto:Token: None 2024-03-04T16:26:19.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.031 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a8fbd873b8a8328-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.033 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.033 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.033 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.033 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.034 DEBUG:boto:Method: GET 2024-03-04T16:26:19.034 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.034 DEBUG:boto:Data: 2024-03-04T16:26:19.034 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.034 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.034 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.034 DEBUG:boto:Params: {} 2024-03-04T16:26:19.034 DEBUG:boto:Token: None 2024-03-04T16:26:19.034 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8ac790f83d1adc7-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.036 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.037 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.037 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.037 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.037 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.037 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.037 DEBUG:boto:Data: 2024-03-04T16:26:19.037 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.038 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.038 DEBUG:boto:Params: {} 2024-03-04T16:26:19.038 DEBUG:boto:Token: None 2024-03-04T16:26:19.038 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tEhdNqlBs7bEIcYntkwYxKOiRFo= 2024-03-04T16:26:19.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tEhdNqlBs7bEIcYntkwYxKOiRFo=', 'Content-Length': '0'} 2024-03-04T16:26:19.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f51a8029e94257b0-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.040 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.040 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.041 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.041 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.041 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.041 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.041 DEBUG:boto:Data: 2024-03-04T16:26:19.041 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.041 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.041 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.041 DEBUG:boto:Params: {} 2024-03-04T16:26:19.041 DEBUG:boto:Token: None 2024-03-04T16:26:19.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.042 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tEhdNqlBs7bEIcYntkwYxKOiRFo= 2024-03-04T16:26:19.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tEhdNqlBs7bEIcYntkwYxKOiRFo=', 'Content-Length': '0'} 2024-03-04T16:26:19.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b6549dea0169057-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:19.044 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.044 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.044 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.044 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.044 DEBUG:boto:Method: GET 2024-03-04T16:26:19.044 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.044 DEBUG:boto:Data: 2024-03-04T16:26:19.044 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.044 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.044 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.045 DEBUG:boto:Params: {} 2024-03-04T16:26:19.045 DEBUG:boto:Token: None 2024-03-04T16:26:19.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.045 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000208d81be524c5f9b-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.049 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.049 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.050 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.050 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.050 DEBUG:boto:Method: GET 2024-03-04T16:26:19.050 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.050 DEBUG:boto:Data: 2024-03-04T16:26:19.050 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.050 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.050 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.050 DEBUG:boto:Params: {} 2024-03-04T16:26:19.050 DEBUG:boto:Token: None 2024-03-04T16:26:19.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.051 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bef4bab96e4db72e-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.052 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.053 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.053 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.053 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.053 DEBUG:boto:Method: GET 2024-03-04T16:26:19.053 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.053 DEBUG:boto:Data: 2024-03-04T16:26:19.053 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.053 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.053 DEBUG:boto:Params: {} 2024-03-04T16:26:19.053 DEBUG:boto:Token: None 2024-03-04T16:26:19.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.054 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c293031ef91b1c08-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.096 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.097 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.097 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.097 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.097 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.097 DEBUG:boto:Method: GET 2024-03-04T16:26:19.097 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.097 DEBUG:boto:Data: 2024-03-04T16:26:19.097 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.097 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.098 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.098 DEBUG:boto:Params: {} 2024-03-04T16:26:19.098 DEBUG:boto:Token: None 2024-03-04T16:26:19.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.098 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a610dcc4a1db15d4-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.101 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.101 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.101 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.101 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.101 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:19.101 DEBUG:boto:Data: 2024-03-04T16:26:19.102 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.102 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.102 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.102 DEBUG:boto:Params: {} 2024-03-04T16:26:19.102 DEBUG:boto:Token: None 2024-03-04T16:26:19.102 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy 2024-03-04T16:26:19.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CpIDrfMLnYUf+S3xlhsWrgaywmE= 2024-03-04T16:26:19.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CpIDrfMLnYUf+S3xlhsWrgaywmE=', 'Content-Length': '0'} 2024-03-04T16:26:19.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091f2e2a9fa34e463-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.105 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.105 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.105 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.105 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:26:19.105 DEBUG:boto:Data: 2024-03-04T16:26:19.105 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.105 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.105 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.105 DEBUG:boto:Params: {} 2024-03-04T16:26:19.105 DEBUG:boto:Token: None 2024-03-04T16:26:19.106 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy 2024-03-04T16:26:19.106 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CpIDrfMLnYUf+S3xlhsWrgaywmE= 2024-03-04T16:26:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CpIDrfMLnYUf+S3xlhsWrgaywmE=', 'Content-Length': '0'} 2024-03-04T16:26:19.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007dadd65f48f49f43-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:19.108 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.108 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.109 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.109 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.109 DEBUG:boto:Method: GET 2024-03-04T16:26:19.109 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.109 DEBUG:boto:Data: 2024-03-04T16:26:19.109 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.109 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.109 DEBUG:boto:Params: {} 2024-03-04T16:26:19.109 DEBUG:boto:Token: None 2024-03-04T16:26:19.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b23d75dcf92cfaa-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.111 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:26:19.112 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:26:19.112 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.112 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.112 DEBUG:boto:Method: GET 2024-03-04T16:26:19.112 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.112 DEBUG:boto:Data: 2024-03-04T16:26:19.112 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.112 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.112 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.112 DEBUG:boto:Params: {} 2024-03-04T16:26:19.113 DEBUG:boto:Token: None 2024-03-04T16:26:19.113 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:26:19.113 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As= 2024-03-04T16:26:19.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+5bUmL7odTOoeMr64w8FQOn66As=', 'Content-Length': '0'} 2024-03-04T16:26:19.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003fbbed1796aa2cb-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.115 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.115 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.115 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.115 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.115 DEBUG:boto:Method: GET 2024-03-04T16:26:19.115 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.115 DEBUG:boto:Data: 2024-03-04T16:26:19.115 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.115 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.115 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.116 DEBUG:boto:Params: {} 2024-03-04T16:26:19.116 DEBUG:boto:Token: None 2024-03-04T16:26:19.116 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.116 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be978fe52674d987-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.119 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.119 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.119 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.119 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.119 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.119 DEBUG:boto:Method: GET 2024-03-04T16:26:19.120 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:26:19.120 DEBUG:boto:Data: 2024-03-04T16:26:19.120 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.120 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.120 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.120 DEBUG:boto:Params: {} 2024-03-04T16:26:19.120 DEBUG:boto:Token: None 2024-03-04T16:26:19.120 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000048b9f9343049fcd-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.123 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:19.123 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.124 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.124 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.124 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.124 DEBUG:boto:Method: GET 2024-03-04T16:26:19.124 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.124 DEBUG:boto:Data: 2024-03-04T16:26:19.124 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.124 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.124 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.124 DEBUG:boto:Params: {} 2024-03-04T16:26:19.124 DEBUG:boto:Token: None 2024-03-04T16:26:19.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cb06f96741ca0db-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.168 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.168 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.168 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.168 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.169 DEBUG:boto:Method: GET 2024-03-04T16:26:19.169 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.169 DEBUG:boto:Data: 2024-03-04T16:26:19.169 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.169 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.169 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.169 DEBUG:boto:Params: {} 2024-03-04T16:26:19.169 DEBUG:boto:Token: None 2024-03-04T16:26:19.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e3be5f563fd7a20-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.172 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.172 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.173 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.173 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.173 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.173 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.173 DEBUG:boto:Data: 2024-03-04T16:26:19.173 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.174 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.174 DEBUG:boto:Params: {} 2024-03-04T16:26:19.174 DEBUG:boto:Token: None 2024-03-04T16:26:19.174 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Iw/B1UUuAK4GU9gl99b6IMziJg= 2024-03-04T16:26:19.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Iw/B1UUuAK4GU9gl99b6IMziJg=', 'Content-Length': '0'} 2024-03-04T16:26:19.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009af99c128706ff5a-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.178 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.179 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.179 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.179 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.179 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.179 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.179 DEBUG:boto:Data: 2024-03-04T16:26:19.179 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.179 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.179 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.179 DEBUG:boto:Params: {} 2024-03-04T16:26:19.179 DEBUG:boto:Token: None 2024-03-04T16:26:19.180 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.180 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Iw/B1UUuAK4GU9gl99b6IMziJg= 2024-03-04T16:26:19.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Iw/B1UUuAK4GU9gl99b6IMziJg=', 'Content-Length': '0'} 2024-03-04T16:26:19.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c9944ce9283ddc3-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:19.182 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.182 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.182 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.182 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.182 DEBUG:boto:Method: GET 2024-03-04T16:26:19.182 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.182 DEBUG:boto:Data: 2024-03-04T16:26:19.183 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.183 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.183 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.183 DEBUG:boto:Params: {} 2024-03-04T16:26:19.183 DEBUG:boto:Token: None 2024-03-04T16:26:19.183 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.184 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042fdd587c1a55afa-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.232 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.232 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.233 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.233 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.233 DEBUG:boto:Method: GET 2024-03-04T16:26:19.233 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.233 DEBUG:boto:Data: 2024-03-04T16:26:19.233 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.233 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.233 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.233 DEBUG:boto:Params: {} 2024-03-04T16:26:19.233 DEBUG:boto:Token: None 2024-03-04T16:26:19.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ba4111bd230a0df-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.276 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.276 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.276 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.277 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.277 DEBUG:boto:Method: GET 2024-03-04T16:26:19.277 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.277 DEBUG:boto:Data: 2024-03-04T16:26:19.277 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.277 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.277 DEBUG:boto:Params: {} 2024-03-04T16:26:19.277 DEBUG:boto:Token: None 2024-03-04T16:26:19.278 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.278 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000818e18f6a9dbab2f-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.280 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.281 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.281 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.281 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.281 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.281 DEBUG:boto:Method: GET 2024-03-04T16:26:19.281 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.281 DEBUG:boto:Data: 2024-03-04T16:26:19.281 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.281 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.281 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.282 DEBUG:boto:Params: {} 2024-03-04T16:26:19.282 DEBUG:boto:Token: None 2024-03-04T16:26:19.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.282 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094989bb58bcd46ed-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.284 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.285 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.285 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.285 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.285 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.285 DEBUG:boto:Data: 2024-03-04T16:26:19.285 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.285 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.286 DEBUG:boto:Params: {} 2024-03-04T16:26:19.286 DEBUG:boto:Token: None 2024-03-04T16:26:19.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:goTlSsPliINvu8t6VSeBULZmVyM= 2024-03-04T16:26:19.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:goTlSsPliINvu8t6VSeBULZmVyM=', 'Content-Length': '0'} 2024-03-04T16:26:19.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ada3fe2d1c7abc0-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.288 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.288 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.289 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.289 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.289 DEBUG:boto:Data: 2024-03-04T16:26:19.289 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.289 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.289 DEBUG:boto:Params: {} 2024-03-04T16:26:19.289 DEBUG:boto:Token: None 2024-03-04T16:26:19.289 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.289 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:goTlSsPliINvu8t6VSeBULZmVyM= 2024-03-04T16:26:19.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:goTlSsPliINvu8t6VSeBULZmVyM=', 'Content-Length': '0'} 2024-03-04T16:26:19.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bb080beb2e70034-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:19.292 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.292 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.292 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.293 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.293 DEBUG:boto:Method: GET 2024-03-04T16:26:19.293 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.293 DEBUG:boto:Data: 2024-03-04T16:26:19.293 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.293 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.293 DEBUG:boto:Params: {} 2024-03-04T16:26:19.293 DEBUG:boto:Token: None 2024-03-04T16:26:19.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003293bc3bb807d485-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.336 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.336 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:26:19.336 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.336 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.337 DEBUG:boto:Method: GET 2024-03-04T16:26:19.337 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.337 DEBUG:boto:Data: 2024-03-04T16:26:19.337 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.337 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.337 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.337 DEBUG:boto:Params: {} 2024-03-04T16:26:19.337 DEBUG:boto:Token: None 2024-03-04T16:26:19.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:26:19.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU= 2024-03-04T16:26:19.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GAYE+XG34EVN0jS/XcSGT/G50PU=', 'Content-Length': '0'} 2024-03-04T16:26:19.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7c4ff20ed826c3a-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.380 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.380 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.381 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.381 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.381 DEBUG:boto:Method: GET 2024-03-04T16:26:19.381 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.381 DEBUG:boto:Data: 2024-03-04T16:26:19.381 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.381 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.381 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.381 DEBUG:boto:Params: {} 2024-03-04T16:26:19.382 DEBUG:boto:Token: None 2024-03-04T16:26:19.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d574c3f67414b12-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.428 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.429 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.429 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.429 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.429 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.429 DEBUG:boto:Method: GET 2024-03-04T16:26:19.429 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:26:19.429 DEBUG:boto:Data: 2024-03-04T16:26:19.429 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.429 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.429 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.430 DEBUG:boto:Params: {} 2024-03-04T16:26:19.430 DEBUG:boto:Token: None 2024-03-04T16:26:19.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f2a7ca7c03d357-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.476 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:19.477 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.477 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.477 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.477 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.477 DEBUG:boto:Method: GET 2024-03-04T16:26:19.477 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.477 DEBUG:boto:Data: 2024-03-04T16:26:19.477 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.478 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.478 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.478 DEBUG:boto:Params: {} 2024-03-04T16:26:19.478 DEBUG:boto:Token: None 2024-03-04T16:26:19.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9c1009355181ecd-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.481 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.481 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.481 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.481 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.481 DEBUG:boto:Method: GET 2024-03-04T16:26:19.481 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.481 DEBUG:boto:Data: 2024-03-04T16:26:19.481 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.482 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.482 DEBUG:boto:Params: {} 2024-03-04T16:26:19.482 DEBUG:boto:Token: None 2024-03-04T16:26:19.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3bcccdd716c4c43-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.484 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.484 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.485 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.485 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.485 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.485 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.485 DEBUG:boto:Data: 2024-03-04T16:26:19.485 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.485 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.485 DEBUG:boto:Params: {} 2024-03-04T16:26:19.485 DEBUG:boto:Token: None 2024-03-04T16:26:19.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/xtDeihh41oivUE2nel3F5IJS+Q= 2024-03-04T16:26:19.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/xtDeihh41oivUE2nel3F5IJS+Q=', 'Content-Length': '0'} 2024-03-04T16:26:19.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4fb9befb189fc45-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.488 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.488 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.488 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.488 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.488 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.488 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.488 DEBUG:boto:Data: 2024-03-04T16:26:19.488 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.488 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.489 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.489 DEBUG:boto:Params: {} 2024-03-04T16:26:19.489 DEBUG:boto:Token: None 2024-03-04T16:26:19.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/xtDeihh41oivUE2nel3F5IJS+Q= 2024-03-04T16:26:19.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/xtDeihh41oivUE2nel3F5IJS+Q=', 'Content-Length': '0'} 2024-03-04T16:26:19.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f932864b63574612-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:19.491 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.491 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.491 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.492 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.492 DEBUG:boto:Method: GET 2024-03-04T16:26:19.492 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.492 DEBUG:boto:Data: 2024-03-04T16:26:19.492 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.492 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.492 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.492 DEBUG:boto:Params: {} 2024-03-04T16:26:19.492 DEBUG:boto:Token: None 2024-03-04T16:26:19.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e92f1e08040087a2-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.540 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.541 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.541 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.541 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.541 DEBUG:boto:Method: GET 2024-03-04T16:26:19.541 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.541 DEBUG:boto:Data: 2024-03-04T16:26:19.541 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.541 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.541 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.541 DEBUG:boto:Params: {} 2024-03-04T16:26:19.542 DEBUG:boto:Token: None 2024-03-04T16:26:19.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be21170c645edf53-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.544 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.544 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.545 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.545 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.545 DEBUG:boto:Method: GET 2024-03-04T16:26:19.545 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.545 DEBUG:boto:Data: 2024-03-04T16:26:19.545 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.545 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.545 DEBUG:boto:Params: {} 2024-03-04T16:26:19.545 DEBUG:boto:Token: None 2024-03-04T16:26:19.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfdbdff5b92d1538-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.548 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.548 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.549 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.549 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.549 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.549 DEBUG:boto:Method: GET 2024-03-04T16:26:19.549 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.549 DEBUG:boto:Data: 2024-03-04T16:26:19.549 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.549 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.549 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.550 DEBUG:boto:Params: {} 2024-03-04T16:26:19.550 DEBUG:boto:Token: None 2024-03-04T16:26:19.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf8d88eecc12b45-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.596 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.596 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.597 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.597 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.597 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.597 DEBUG:boto:Data: 2024-03-04T16:26:19.597 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.597 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.597 DEBUG:boto:Params: {} 2024-03-04T16:26:19.597 DEBUG:boto:Token: None 2024-03-04T16:26:19.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zmMaQ5CihncaTRvf3Kct9Pnxo8= 2024-03-04T16:26:19.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zmMaQ5CihncaTRvf3Kct9Pnxo8=', 'Content-Length': '0'} 2024-03-04T16:26:19.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000139ba1044e622e90-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.600 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.600 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.600 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.600 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.600 DEBUG:boto:Data: 2024-03-04T16:26:19.600 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.600 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.600 DEBUG:boto:Params: {} 2024-03-04T16:26:19.601 DEBUG:boto:Token: None 2024-03-04T16:26:19.601 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zmMaQ5CihncaTRvf3Kct9Pnxo8= 2024-03-04T16:26:19.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zmMaQ5CihncaTRvf3Kct9Pnxo8=', 'Content-Length': '0'} 2024-03-04T16:26:19.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b96b8a50a0e039cc-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:19.604 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.604 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.604 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.604 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.605 DEBUG:boto:Method: GET 2024-03-04T16:26:19.605 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.605 DEBUG:boto:Data: 2024-03-04T16:26:19.605 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.605 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.605 DEBUG:boto:Params: {} 2024-03-04T16:26:19.605 DEBUG:boto:Token: None 2024-03-04T16:26:19.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e56ab62b6c14a888-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.648 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.649 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:26:19.649 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.649 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.649 DEBUG:boto:Method: GET 2024-03-04T16:26:19.649 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.649 DEBUG:boto:Data: 2024-03-04T16:26:19.650 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.650 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.650 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.650 DEBUG:boto:Params: {} 2024-03-04T16:26:19.650 DEBUG:boto:Token: None 2024-03-04T16:26:19.651 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:26:19.651 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4= 2024-03-04T16:26:19.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B/OedVY147dPM7MvfVYVqO3O8o4=', 'Content-Length': '0'} 2024-03-04T16:26:19.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1b03c936bf4751f-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.654 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.654 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.654 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.654 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.654 DEBUG:boto:Method: GET 2024-03-04T16:26:19.654 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.655 DEBUG:boto:Data: 2024-03-04T16:26:19.655 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.655 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.655 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.655 DEBUG:boto:Params: {} 2024-03-04T16:26:19.655 DEBUG:boto:Token: None 2024-03-04T16:26:19.656 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.656 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c0773b10a6010a-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.658 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.659 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.659 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.659 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.659 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.659 DEBUG:boto:Method: GET 2024-03-04T16:26:19.659 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:26:19.660 DEBUG:boto:Data: 2024-03-04T16:26:19.660 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.660 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.660 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.660 DEBUG:boto:Params: {} 2024-03-04T16:26:19.660 DEBUG:boto:Token: None 2024-03-04T16:26:19.660 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a824be1567eb086-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.664 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:19.664 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.664 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.664 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.664 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.664 DEBUG:boto:Method: GET 2024-03-04T16:26:19.665 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.665 DEBUG:boto:Data: 2024-03-04T16:26:19.665 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.665 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.665 DEBUG:boto:Params: {} 2024-03-04T16:26:19.665 DEBUG:boto:Token: None 2024-03-04T16:26:19.665 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.665 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b0ce592ab365bf4-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.667 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.668 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.668 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.668 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.668 DEBUG:boto:Method: GET 2024-03-04T16:26:19.668 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.668 DEBUG:boto:Data: 2024-03-04T16:26:19.668 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.668 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.668 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.668 DEBUG:boto:Params: {} 2024-03-04T16:26:19.669 DEBUG:boto:Token: None 2024-03-04T16:26:19.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000697a09eff06135f-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.716 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.716 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.716 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.717 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.717 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.717 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.717 DEBUG:boto:Data: 2024-03-04T16:26:19.717 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.717 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.717 DEBUG:boto:Params: {} 2024-03-04T16:26:19.717 DEBUG:boto:Token: None 2024-03-04T16:26:19.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K5LAF9JCkDtfkxjujB8Q1yiyRcQ= 2024-03-04T16:26:19.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K5LAF9JCkDtfkxjujB8Q1yiyRcQ=', 'Content-Length': '0'} 2024-03-04T16:26:19.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b986087d371ccdb-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.720 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.720 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.720 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.720 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.720 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.721 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.721 DEBUG:boto:Data: 2024-03-04T16:26:19.721 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.721 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.721 DEBUG:boto:Params: {} 2024-03-04T16:26:19.721 DEBUG:boto:Token: None 2024-03-04T16:26:19.721 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K5LAF9JCkDtfkxjujB8Q1yiyRcQ= 2024-03-04T16:26:19.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K5LAF9JCkDtfkxjujB8Q1yiyRcQ=', 'Content-Length': '0'} 2024-03-04T16:26:19.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000652dec3bbf2b725-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:19.724 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.724 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.724 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.724 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.724 DEBUG:boto:Method: GET 2024-03-04T16:26:19.725 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.725 DEBUG:boto:Data: 2024-03-04T16:26:19.725 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.725 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.725 DEBUG:boto:Params: {} 2024-03-04T16:26:19.725 DEBUG:boto:Token: None 2024-03-04T16:26:19.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.725 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adddcc4241a5909b-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.727 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.728 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.728 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.728 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.728 DEBUG:boto:Method: GET 2024-03-04T16:26:19.728 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.728 DEBUG:boto:Data: 2024-03-04T16:26:19.728 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.728 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.728 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.728 DEBUG:boto:Params: {} 2024-03-04T16:26:19.728 DEBUG:boto:Token: None 2024-03-04T16:26:19.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.729 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e143766ea3208dd-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.772 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.772 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.772 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.773 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.773 DEBUG:boto:Method: GET 2024-03-04T16:26:19.773 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.773 DEBUG:boto:Data: 2024-03-04T16:26:19.773 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.773 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.773 DEBUG:boto:Params: {} 2024-03-04T16:26:19.773 DEBUG:boto:Token: None 2024-03-04T16:26:19.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c207a4967f71ee84-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.776 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.777 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.777 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.777 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.777 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.777 DEBUG:boto:Method: GET 2024-03-04T16:26:19.777 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.777 DEBUG:boto:Data: 2024-03-04T16:26:19.777 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.777 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.777 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.778 DEBUG:boto:Params: {} 2024-03-04T16:26:19.778 DEBUG:boto:Token: None 2024-03-04T16:26:19.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065dfbf64f19fb2ef-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.781 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.781 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.781 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.781 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.781 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.782 DEBUG:boto:Data: 2024-03-04T16:26:19.782 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.782 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.782 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.782 DEBUG:boto:Params: {} 2024-03-04T16:26:19.782 DEBUG:boto:Token: None 2024-03-04T16:26:19.782 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9EVO9bV7Ww2lKoRw/gFQaXefzak= 2024-03-04T16:26:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9EVO9bV7Ww2lKoRw/gFQaXefzak=', 'Content-Length': '0'} 2024-03-04T16:26:19.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9c8d39b58583725-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.784 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.784 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.784 DEBUG:boto:Method: HEAD 2024-03-04T16:26:19.785 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.785 DEBUG:boto:Data: 2024-03-04T16:26:19.785 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.785 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.785 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.785 DEBUG:boto:Params: {} 2024-03-04T16:26:19.785 DEBUG:boto:Token: None 2024-03-04T16:26:19.785 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9EVO9bV7Ww2lKoRw/gFQaXefzak= 2024-03-04T16:26:19.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9EVO9bV7Ww2lKoRw/gFQaXefzak=', 'Content-Length': '0'} 2024-03-04T16:26:19.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ad9b0d5286abb68-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:19.788 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.788 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.788 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.788 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.788 DEBUG:boto:Method: GET 2024-03-04T16:26:19.789 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.789 DEBUG:boto:Data: 2024-03-04T16:26:19.789 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.789 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.789 DEBUG:boto:Params: {} 2024-03-04T16:26:19.789 DEBUG:boto:Token: None 2024-03-04T16:26:19.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a20db8b4dd7e7516-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.832 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.832 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:26:19.832 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.832 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.832 DEBUG:boto:Method: GET 2024-03-04T16:26:19.833 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.833 DEBUG:boto:Data: 2024-03-04T16:26:19.833 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.833 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.833 DEBUG:boto:Params: {} 2024-03-04T16:26:19.833 DEBUG:boto:Token: None 2024-03-04T16:26:19.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:26:19.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI= 2024-03-04T16:26:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z46+iI3jkrlnGEP4wPYrh0exQWI=', 'Content-Length': '0'} 2024-03-04T16:26:19.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de8944e291f1875f-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.876 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.876 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.876 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.877 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.877 DEBUG:boto:Method: GET 2024-03-04T16:26:19.877 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.877 DEBUG:boto:Data: 2024-03-04T16:26:19.877 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.877 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:19.877 DEBUG:boto:Port: 8001 2024-03-04T16:26:19.877 DEBUG:boto:Params: {} 2024-03-04T16:26:19.877 DEBUG:boto:Token: None 2024-03-04T16:26:19.877 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006afd10c26cbc15b5-0065e5f62b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.880 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.881 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:26:19.881 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:26:19.881 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.881 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.881 DEBUG:boto:Method: GET 2024-03-04T16:26:19.881 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:26:19.881 DEBUG:boto:Data: 2024-03-04T16:26:19.881 DEBUG:boto:Headers: {} 2024-03-04T16:26:19.882 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:19.882 DEBUG:boto:Port: 8000 2024-03-04T16:26:19.882 DEBUG:boto:Params: {} 2024-03-04T16:26:19.882 DEBUG:boto:Token: None 2024-03-04T16:26:19.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:19 GMT /sbnvju-19/ 2024-03-04T16:26:19.883 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68= 2024-03-04T16:26:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ugFkyI62OjAISa87xYJjl3E+u68=', 'Content-Length': '0'} 2024-03-04T16:26:19.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe17919e82580a15-0065e5f62b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:19.932 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:19.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone3, test-zone2} 2024-03-04T16:26:19.933 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-20 2024-03-04T16:26:19.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:19.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:19.933 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:19.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:19.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:20.049 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:20.048+0000 7fb969e17800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:20.049 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000006.34.11, , , , , ] 2024-03-04T16:26:20.049 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:25.050 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:25.050+0000 7fb969e17800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:25.050 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000006.34.11, , , , , ] 2024-03-04T16:26:25.050 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:30.051 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:30.051+0000 7fb969e17800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:30.052 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000006.34.11, , , , , ] 2024-03-04T16:26:30.052 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:35.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:35.054+0000 7fb969e17800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:35.053 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000006.34.11, , , , , ] 2024-03-04T16:26:35.053 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:40.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:40.054+0000 7fb969e17800 1 bucket sync caught up with source: 2024-03-04T16:26:40.055 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:40.055 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:40.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:40.054+0000 7fb969e17800 0 bucket checkpoint complete 2024-03-04T16:26:40.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-20 2024-03-04T16:26:40.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.084 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.194+0000 7f338efa2800 1 bucket sync caught up with source: 2024-03-04T16:26:40.194 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:40.194 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000010.36.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:26:40.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.194+0000 7f338efa2800 0 bucket checkpoint complete 2024-03-04T16:26:40.213 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-20 2024-03-04T16:26:40.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.213 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:40.267 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:40.267 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:40.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.318+0000 7ff3881af800 1 bucket sync caught up with source: 2024-03-04T16:26:40.318 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:26:40.318 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:26:40.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.318+0000 7ff3881af800 0 bucket checkpoint complete 2024-03-04T16:26:40.337 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-20 2024-03-04T16:26:40.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.338 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:40.393 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:40.393 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.443+0000 7f3b03024800 1 bucket sync caught up with source: 2024-03-04T16:26:40.443 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:26:40.444 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.22.10, , 00000000010.31.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:26:40.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.443+0000 7f3b03024800 0 bucket checkpoint complete 2024-03-04T16:26:40.462 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-20 2024-03-04T16:26:40.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.462 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:40.516 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:40.517 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:40.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.566+0000 7f1ab6267800 1 bucket sync caught up with source: 2024-03-04T16:26:40.569 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.20.10, , 00000000003.21.10, , 00000000014.60.10, , , , , ] 2024-03-04T16:26:40.569 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.20.10, , 00000000003.21.10, , 00000000014.60.10, , , , , ] 2024-03-04T16:26:40.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:26:40.566+0000 7f1ab6267800 0 bucket checkpoint complete 2024-03-04T16:26:40.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-20 2024-03-04T16:26:40.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:40.586 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:40.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:40.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:40.699+0000 7fa1f8a24800 1 bucket sync caught up with source: 2024-03-04T16:26:40.700 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.20.10, , 00000000003.21.10, , 00000000014.60.10, , , , , ] 2024-03-04T16:26:40.700 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.20.10, , 00000000003.21.10, , 00000000014.60.10, , , , , ] 2024-03-04T16:26:40.701 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:40.699+0000 7fa1f8a24800 0 bucket checkpoint complete 2024-03-04T16:26:40.717 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone2, test-zone1} 2024-03-04T16:26:40.718 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.718 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.718 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.718 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:40.718 DEBUG:boto:Data: 2024-03-04T16:26:40.718 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.718 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.718 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.718 DEBUG:boto:Params: {} 2024-03-04T16:26:40.719 DEBUG:boto:Token: None 2024-03-04T16:26:40.719 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.719 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+vIS0/P/cv6iufgxNQxZ/ucZIJo= 2024-03-04T16:26:40.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+vIS0/P/cv6iufgxNQxZ/ucZIJo=', 'Content-Length': '0'} 2024-03-04T16:26:40.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000accabf08325f0ebd-0065e5f640-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.722 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.722 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.722 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.722 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:40.723 DEBUG:boto:Data: 2024-03-04T16:26:40.723 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.723 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.723 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.723 DEBUG:boto:Params: {} 2024-03-04T16:26:40.723 DEBUG:boto:Token: None 2024-03-04T16:26:40.723 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.723 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+vIS0/P/cv6iufgxNQxZ/ucZIJo= 2024-03-04T16:26:40.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+vIS0/P/cv6iufgxNQxZ/ucZIJo=', 'Content-Length': '0'} 2024-03-04T16:26:40.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f496a887cbf0379-0065e5f640-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:40.726 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.726 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.726 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.726 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.726 DEBUG:boto:Method: GET 2024-03-04T16:26:40.726 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.726 DEBUG:boto:Data: 2024-03-04T16:26:40.726 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.727 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.727 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.727 DEBUG:boto:Params: {} 2024-03-04T16:26:40.727 DEBUG:boto:Token: None 2024-03-04T16:26:40.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/?versions 2024-03-04T16:26:40.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE= 2024-03-04T16:26:40.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE=', 'Content-Length': '0'} 2024-03-04T16:26:40.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a59117251c04b5-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.730 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:40.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:40.731 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.731 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.731 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.732 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.732 DEBUG:boto:Method: GET 2024-03-04T16:26:40.732 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.732 DEBUG:boto:Data: 2024-03-04T16:26:40.732 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.732 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.732 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.732 DEBUG:boto:Params: {} 2024-03-04T16:26:40.732 DEBUG:boto:Token: None 2024-03-04T16:26:40.732 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/?versions 2024-03-04T16:26:40.733 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE= 2024-03-04T16:26:40.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE=', 'Content-Length': '0'} 2024-03-04T16:26:40.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041143d3f2b323a8f-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.735 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:40.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:40.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:40.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:40.736 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.736 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.736 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.736 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.736 DEBUG:boto:Method: GET 2024-03-04T16:26:40.736 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.736 DEBUG:boto:Data: 2024-03-04T16:26:40.736 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.737 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.737 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.737 DEBUG:boto:Params: {} 2024-03-04T16:26:40.737 DEBUG:boto:Token: None 2024-03-04T16:26:40.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/?versions 2024-03-04T16:26:40.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE= 2024-03-04T16:26:40.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE=', 'Content-Length': '0'} 2024-03-04T16:26:40.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007044e806b6a820c7-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.740 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.740 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.740 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.740 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.740 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.741 DEBUG:boto:Method: GET 2024-03-04T16:26:40.741 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:40.741 DEBUG:boto:Data: 2024-03-04T16:26:40.741 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.741 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.741 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.741 DEBUG:boto:Params: {} 2024-03-04T16:26:40.741 DEBUG:boto:Token: None 2024-03-04T16:26:40.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/?versions 2024-03-04T16:26:40.741 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE= 2024-03-04T16:26:40.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E/1sNV250FzidkbCkQBVVh/7mbE=', 'Content-Length': '0'} 2024-03-04T16:26:40.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58ae59c8da83c24-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.744 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:40.745 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.745 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.745 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.745 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.745 DEBUG:boto:Method: GET 2024-03-04T16:26:40.745 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.745 DEBUG:boto:Data: 2024-03-04T16:26:40.745 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.745 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.745 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.745 DEBUG:boto:Params: {} 2024-03-04T16:26:40.746 DEBUG:boto:Token: None 2024-03-04T16:26:40.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f51bf42e280f424-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.748 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.748 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.748 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.748 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.748 DEBUG:boto:Method: GET 2024-03-04T16:26:40.748 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.748 DEBUG:boto:Data: 2024-03-04T16:26:40.748 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.748 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.749 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.749 DEBUG:boto:Params: {} 2024-03-04T16:26:40.749 DEBUG:boto:Token: None 2024-03-04T16:26:40.749 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.749 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001deaa2b83717d32b-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.751 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.751 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.751 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.751 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.751 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.752 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.752 DEBUG:boto:Data: 2024-03-04T16:26:40.752 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.752 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.752 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.752 DEBUG:boto:Params: {} 2024-03-04T16:26:40.752 DEBUG:boto:Token: None 2024-03-04T16:26:40.752 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.752 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OGg6+BIqDZ5oO69N5oPDC/XeNgA= 2024-03-04T16:26:40.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OGg6+BIqDZ5oO69N5oPDC/XeNgA=', 'Content-Length': '0'} 2024-03-04T16:26:40.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f949aa11c156310b-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.754 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.754 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.754 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.754 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.755 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.755 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.755 DEBUG:boto:Data: 2024-03-04T16:26:40.755 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.755 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.755 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.755 DEBUG:boto:Params: {} 2024-03-04T16:26:40.755 DEBUG:boto:Token: None 2024-03-04T16:26:40.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OGg6+BIqDZ5oO69N5oPDC/XeNgA= 2024-03-04T16:26:40.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OGg6+BIqDZ5oO69N5oPDC/XeNgA=', 'Content-Length': '0'} 2024-03-04T16:26:40.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000daaa7114ed25dfda-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:40.757 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.757 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.758 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.758 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.758 DEBUG:boto:Method: GET 2024-03-04T16:26:40.758 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.758 DEBUG:boto:Data: 2024-03-04T16:26:40.758 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.758 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.758 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.758 DEBUG:boto:Params: {} 2024-03-04T16:26:40.758 DEBUG:boto:Token: None 2024-03-04T16:26:40.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.759 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071048fd780ee96f6-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.761 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.761 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.761 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.761 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.761 DEBUG:boto:Method: GET 2024-03-04T16:26:40.761 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.761 DEBUG:boto:Data: 2024-03-04T16:26:40.761 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.762 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.762 DEBUG:boto:Params: {} 2024-03-04T16:26:40.762 DEBUG:boto:Token: None 2024-03-04T16:26:40.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.762 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d75881968a02ccd-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.764 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.764 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.764 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.764 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.765 DEBUG:boto:Method: GET 2024-03-04T16:26:40.765 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.765 DEBUG:boto:Data: 2024-03-04T16:26:40.765 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.765 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.765 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.765 DEBUG:boto:Params: {} 2024-03-04T16:26:40.765 DEBUG:boto:Token: None 2024-03-04T16:26:40.765 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.765 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d8480ee36475e4f-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.768 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.768 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.768 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.768 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.768 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.768 DEBUG:boto:Method: GET 2024-03-04T16:26:40.769 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.769 DEBUG:boto:Data: 2024-03-04T16:26:40.769 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.769 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.769 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.769 DEBUG:boto:Params: {} 2024-03-04T16:26:40.769 DEBUG:boto:Token: None 2024-03-04T16:26:40.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.769 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006978179070c775b2-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.772 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.772 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.772 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.772 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.772 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:40.773 DEBUG:boto:Data: 2024-03-04T16:26:40.773 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.773 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.773 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.773 DEBUG:boto:Params: {} 2024-03-04T16:26:40.773 DEBUG:boto:Token: None 2024-03-04T16:26:40.773 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy 2024-03-04T16:26:40.773 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mGvXMVC1RRMn6WUsKWIwdai22TU= 2024-03-04T16:26:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mGvXMVC1RRMn6WUsKWIwdai22TU=', 'Content-Length': '0'} 2024-03-04T16:26:40.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007560befb6fb3a7d7-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.776 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.776 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.776 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.776 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:40.776 DEBUG:boto:Data: 2024-03-04T16:26:40.776 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.776 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.776 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.776 DEBUG:boto:Params: {} 2024-03-04T16:26:40.776 DEBUG:boto:Token: None 2024-03-04T16:26:40.777 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy 2024-03-04T16:26:40.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mGvXMVC1RRMn6WUsKWIwdai22TU= 2024-03-04T16:26:40.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mGvXMVC1RRMn6WUsKWIwdai22TU=', 'Content-Length': '0'} 2024-03-04T16:26:40.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091911d9bd61bc3f3-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:40.779 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.779 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.779 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.779 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.779 DEBUG:boto:Method: GET 2024-03-04T16:26:40.779 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.779 DEBUG:boto:Data: 2024-03-04T16:26:40.779 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.779 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.780 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.780 DEBUG:boto:Params: {} 2024-03-04T16:26:40.780 DEBUG:boto:Token: None 2024-03-04T16:26:40.780 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.780 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df0fe2c81ba79cc0-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.782 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:40.782 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:40.782 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.782 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.782 DEBUG:boto:Method: GET 2024-03-04T16:26:40.782 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.783 DEBUG:boto:Data: 2024-03-04T16:26:40.783 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.783 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.783 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.783 DEBUG:boto:Params: {} 2024-03-04T16:26:40.783 DEBUG:boto:Token: None 2024-03-04T16:26:40.783 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:40.783 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE= 2024-03-04T16:26:40.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fo2+MmkARYkmm0LsWlQmovMAtAE=', 'Content-Length': '0'} 2024-03-04T16:26:40.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e070a6160590fcb6-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.785 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.785 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.785 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.785 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.785 DEBUG:boto:Method: GET 2024-03-04T16:26:40.785 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.786 DEBUG:boto:Data: 2024-03-04T16:26:40.786 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.786 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.786 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.786 DEBUG:boto:Params: {} 2024-03-04T16:26:40.786 DEBUG:boto:Token: None 2024-03-04T16:26:40.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043389321e1a5f7ba-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.788 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.789 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.789 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.789 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.789 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.789 DEBUG:boto:Method: GET 2024-03-04T16:26:40.789 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:40.789 DEBUG:boto:Data: 2024-03-04T16:26:40.789 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.790 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.790 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.790 DEBUG:boto:Params: {} 2024-03-04T16:26:40.790 DEBUG:boto:Token: None 2024-03-04T16:26:40.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9ea3da69b6fa4c-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.792 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:40.793 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.793 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.793 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.793 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.793 DEBUG:boto:Method: GET 2024-03-04T16:26:40.793 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.793 DEBUG:boto:Data: 2024-03-04T16:26:40.793 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.793 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.793 DEBUG:boto:Params: {} 2024-03-04T16:26:40.794 DEBUG:boto:Token: None 2024-03-04T16:26:40.794 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006854d53f11a69478-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.840 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.840 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.840 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.840 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.841 DEBUG:boto:Method: GET 2024-03-04T16:26:40.841 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.841 DEBUG:boto:Data: 2024-03-04T16:26:40.841 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.841 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.841 DEBUG:boto:Params: {} 2024-03-04T16:26:40.841 DEBUG:boto:Token: None 2024-03-04T16:26:40.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044e542e2bb808a3c-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.843 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.844 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.844 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.844 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.844 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.844 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.844 DEBUG:boto:Data: 2024-03-04T16:26:40.844 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.844 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.844 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.844 DEBUG:boto:Params: {} 2024-03-04T16:26:40.845 DEBUG:boto:Token: None 2024-03-04T16:26:40.845 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L5l9GjHY+IXSLXVK5GbenEl75ds= 2024-03-04T16:26:40.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L5l9GjHY+IXSLXVK5GbenEl75ds=', 'Content-Length': '0'} 2024-03-04T16:26:40.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030cde37676ea191c-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.847 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.847 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.847 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.847 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.847 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.847 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.847 DEBUG:boto:Data: 2024-03-04T16:26:40.847 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.848 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.848 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.848 DEBUG:boto:Params: {} 2024-03-04T16:26:40.848 DEBUG:boto:Token: None 2024-03-04T16:26:40.848 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.848 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L5l9GjHY+IXSLXVK5GbenEl75ds= 2024-03-04T16:26:40.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L5l9GjHY+IXSLXVK5GbenEl75ds=', 'Content-Length': '0'} 2024-03-04T16:26:40.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003766356a50e1ac68-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:40.850 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.850 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.850 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.850 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.850 DEBUG:boto:Method: GET 2024-03-04T16:26:40.851 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.851 DEBUG:boto:Data: 2024-03-04T16:26:40.851 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.851 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.851 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.851 DEBUG:boto:Params: {} 2024-03-04T16:26:40.851 DEBUG:boto:Token: None 2024-03-04T16:26:40.851 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.851 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600be53fd9c16d9d-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.896 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.896 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.896 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.896 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.897 DEBUG:boto:Method: GET 2024-03-04T16:26:40.897 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.897 DEBUG:boto:Data: 2024-03-04T16:26:40.897 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.897 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.897 DEBUG:boto:Params: {} 2024-03-04T16:26:40.897 DEBUG:boto:Token: None 2024-03-04T16:26:40.897 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.897 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e6d225bc42173dd-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.900 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.900 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.900 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.900 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.900 DEBUG:boto:Method: GET 2024-03-04T16:26:40.900 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.900 DEBUG:boto:Data: 2024-03-04T16:26:40.900 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.900 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.900 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.901 DEBUG:boto:Params: {} 2024-03-04T16:26:40.901 DEBUG:boto:Token: None 2024-03-04T16:26:40.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.901 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f4b3941ca14cf5-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.904 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.904 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.904 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.904 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.904 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.905 DEBUG:boto:Method: GET 2024-03-04T16:26:40.905 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.905 DEBUG:boto:Data: 2024-03-04T16:26:40.905 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.905 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.905 DEBUG:boto:Params: {} 2024-03-04T16:26:40.905 DEBUG:boto:Token: None 2024-03-04T16:26:40.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.905 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef541f113a41620-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.908 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:40.908 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.908 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.908 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.908 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.908 DEBUG:boto:Data: 2024-03-04T16:26:40.908 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.909 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.909 DEBUG:boto:Params: {} 2024-03-04T16:26:40.909 DEBUG:boto:Token: None 2024-03-04T16:26:40.909 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.909 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:99+Hk58cTC1rNYkeIM8kS4Lq/sk= 2024-03-04T16:26:40.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:99+Hk58cTC1rNYkeIM8kS4Lq/sk=', 'Content-Length': '0'} 2024-03-04T16:26:40.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4fc10474d5e61f-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.911 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.912 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.912 DEBUG:boto:Method: HEAD 2024-03-04T16:26:40.912 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.912 DEBUG:boto:Data: 2024-03-04T16:26:40.912 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.912 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.912 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.912 DEBUG:boto:Params: {} 2024-03-04T16:26:40.912 DEBUG:boto:Token: None 2024-03-04T16:26:40.912 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:99+Hk58cTC1rNYkeIM8kS4Lq/sk= 2024-03-04T16:26:40.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:99+Hk58cTC1rNYkeIM8kS4Lq/sk=', 'Content-Length': '0'} 2024-03-04T16:26:40.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045e380fb1ec507d6-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:40.915 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.915 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.915 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.915 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.916 DEBUG:boto:Method: GET 2024-03-04T16:26:40.916 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.916 DEBUG:boto:Data: 2024-03-04T16:26:40.916 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.916 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.916 DEBUG:boto:Params: {} 2024-03-04T16:26:40.916 DEBUG:boto:Token: None 2024-03-04T16:26:40.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.917 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e0a8331102eae76-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.964 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.964 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:40.964 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.964 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.965 DEBUG:boto:Method: GET 2024-03-04T16:26:40.965 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.965 DEBUG:boto:Data: 2024-03-04T16:26:40.965 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.965 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:40.965 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.965 DEBUG:boto:Params: {} 2024-03-04T16:26:40.965 DEBUG:boto:Token: None 2024-03-04T16:26:40.965 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:40.965 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q= 2024-03-04T16:26:40.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpEjUShK5ZfXB4ZmW47EPb8XR9Q=', 'Content-Length': '0'} 2024-03-04T16:26:40.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3c352f129eab65d-0065e5f640-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:40.968 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:40.968 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:40.968 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.968 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.968 DEBUG:boto:Method: GET 2024-03-04T16:26:40.968 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:40.968 DEBUG:boto:Data: 2024-03-04T16:26:40.968 DEBUG:boto:Headers: {} 2024-03-04T16:26:40.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:40.969 DEBUG:boto:Port: 8000 2024-03-04T16:26:40.969 DEBUG:boto:Params: {} 2024-03-04T16:26:40.969 DEBUG:boto:Token: None 2024-03-04T16:26:40.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:40 GMT /sbnvju-20/ 2024-03-04T16:26:40.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4= 2024-03-04T16:26:40.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gYB6gd1VjjdWRfn/doGrJASjPu4=', 'Content-Length': '0'} 2024-03-04T16:26:40.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da806876dea8feb-0065e5f640-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.012 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.012 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.013 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.013 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.013 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.013 DEBUG:boto:Method: GET 2024-03-04T16:26:41.013 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.013 DEBUG:boto:Data: 2024-03-04T16:26:41.013 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.013 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.013 DEBUG:boto:Params: {} 2024-03-04T16:26:41.014 DEBUG:boto:Token: None 2024-03-04T16:26:41.014 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b956b25b70572c3f-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.016 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:41.017 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.017 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.017 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.017 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.017 DEBUG:boto:Method: GET 2024-03-04T16:26:41.017 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.017 DEBUG:boto:Data: 2024-03-04T16:26:41.018 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.018 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.018 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.018 DEBUG:boto:Params: {} 2024-03-04T16:26:41.018 DEBUG:boto:Token: None 2024-03-04T16:26:41.018 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.018 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c213699c08c0c06-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.020 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.021 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.021 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.021 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.021 DEBUG:boto:Method: GET 2024-03-04T16:26:41.021 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.021 DEBUG:boto:Data: 2024-03-04T16:26:41.021 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.021 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.021 DEBUG:boto:Params: {} 2024-03-04T16:26:41.021 DEBUG:boto:Token: None 2024-03-04T16:26:41.022 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066cd445e41538edc-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.064 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.064 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.064 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.064 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.065 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.065 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.065 DEBUG:boto:Data: 2024-03-04T16:26:41.065 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.065 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.065 DEBUG:boto:Params: {} 2024-03-04T16:26:41.065 DEBUG:boto:Token: None 2024-03-04T16:26:41.065 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I= 2024-03-04T16:26:41.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I=', 'Content-Length': '0'} 2024-03-04T16:26:41.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082dc14bcf23cc5c1-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.067 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.067 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.068 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.068 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.068 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.068 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.068 DEBUG:boto:Data: 2024-03-04T16:26:41.068 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.068 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.068 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.068 DEBUG:boto:Params: {} 2024-03-04T16:26:41.068 DEBUG:boto:Token: None 2024-03-04T16:26:41.069 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I= 2024-03-04T16:26:41.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I=', 'Content-Length': '0'} 2024-03-04T16:26:41.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000928f9c1d5a689ae7-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:41.071 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.071 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.071 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.072 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.072 DEBUG:boto:Method: GET 2024-03-04T16:26:41.072 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.072 DEBUG:boto:Data: 2024-03-04T16:26:41.072 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.072 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.072 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.072 DEBUG:boto:Params: {} 2024-03-04T16:26:41.072 DEBUG:boto:Token: None 2024-03-04T16:26:41.072 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.073 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044796467e979493f-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.075 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.075 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.075 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.075 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.075 DEBUG:boto:Method: GET 2024-03-04T16:26:41.075 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.075 DEBUG:boto:Data: 2024-03-04T16:26:41.075 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.075 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.075 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.076 DEBUG:boto:Params: {} 2024-03-04T16:26:41.076 DEBUG:boto:Token: None 2024-03-04T16:26:41.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.076 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e82b49574327db9-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.120 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.120 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.121 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.121 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.121 DEBUG:boto:Method: GET 2024-03-04T16:26:41.121 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.121 DEBUG:boto:Data: 2024-03-04T16:26:41.121 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.121 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.121 DEBUG:boto:Params: {} 2024-03-04T16:26:41.121 DEBUG:boto:Token: None 2024-03-04T16:26:41.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e75de0b9eb4c18d-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.124 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.125 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.125 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.125 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.125 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.125 DEBUG:boto:Method: GET 2024-03-04T16:26:41.125 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.125 DEBUG:boto:Data: 2024-03-04T16:26:41.125 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.126 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.126 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.126 DEBUG:boto:Params: {} 2024-03-04T16:26:41.126 DEBUG:boto:Token: None 2024-03-04T16:26:41.126 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080293ff4a7a8e92d-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.129 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.129 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.129 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.129 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.129 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.129 DEBUG:boto:Data: 2024-03-04T16:26:41.130 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.130 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.130 DEBUG:boto:Params: {} 2024-03-04T16:26:41.130 DEBUG:boto:Token: None 2024-03-04T16:26:41.130 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ufQFusYyEoqbpiGPXpePTjyx0CY= 2024-03-04T16:26:41.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ufQFusYyEoqbpiGPXpePTjyx0CY=', 'Content-Length': '0'} 2024-03-04T16:26:41.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce266f9fc23e468c-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.132 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.133 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.133 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.133 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.133 DEBUG:boto:Data: 2024-03-04T16:26:41.133 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.133 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.133 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.133 DEBUG:boto:Params: {} 2024-03-04T16:26:41.133 DEBUG:boto:Token: None 2024-03-04T16:26:41.133 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.134 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ufQFusYyEoqbpiGPXpePTjyx0CY= 2024-03-04T16:26:41.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ufQFusYyEoqbpiGPXpePTjyx0CY=', 'Content-Length': '0'} 2024-03-04T16:26:41.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063f5ccfaacf31eaf-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:41.136 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.136 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.136 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.136 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.136 DEBUG:boto:Method: GET 2024-03-04T16:26:41.136 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.136 DEBUG:boto:Data: 2024-03-04T16:26:41.137 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.137 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.137 DEBUG:boto:Params: {} 2024-03-04T16:26:41.137 DEBUG:boto:Token: None 2024-03-04T16:26:41.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004335d7aefaf0311f-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.180 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.181 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.181 DEBUG:boto:Method: GET 2024-03-04T16:26:41.181 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.181 DEBUG:boto:Data: 2024-03-04T16:26:41.181 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.181 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.181 DEBUG:boto:Params: {} 2024-03-04T16:26:41.181 DEBUG:boto:Token: None 2024-03-04T16:26:41.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5deed4885e06e61-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.224 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.224 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.224 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.224 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.225 DEBUG:boto:Method: GET 2024-03-04T16:26:41.225 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.225 DEBUG:boto:Data: 2024-03-04T16:26:41.225 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.225 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.225 DEBUG:boto:Params: {} 2024-03-04T16:26:41.225 DEBUG:boto:Token: None 2024-03-04T16:26:41.225 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.225 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c6a4effbd64ca3-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.228 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.229 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.229 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.229 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.229 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.229 DEBUG:boto:Method: GET 2024-03-04T16:26:41.229 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:41.229 DEBUG:boto:Data: 2024-03-04T16:26:41.229 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.229 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.229 DEBUG:boto:Params: {} 2024-03-04T16:26:41.230 DEBUG:boto:Token: None 2024-03-04T16:26:41.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921f073f1dafb6a2-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.233 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:41.233 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.233 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.233 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.233 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.233 DEBUG:boto:Method: GET 2024-03-04T16:26:41.233 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.234 DEBUG:boto:Data: 2024-03-04T16:26:41.234 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.234 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.234 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.234 DEBUG:boto:Params: {} 2024-03-04T16:26:41.234 DEBUG:boto:Token: None 2024-03-04T16:26:41.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a027c2a5241b6429-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.237 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.237 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.237 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.237 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.237 DEBUG:boto:Method: GET 2024-03-04T16:26:41.237 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.237 DEBUG:boto:Data: 2024-03-04T16:26:41.237 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.237 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.237 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.237 DEBUG:boto:Params: {} 2024-03-04T16:26:41.238 DEBUG:boto:Token: None 2024-03-04T16:26:41.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.238 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066275406275f0cc2-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.280 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.280 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.280 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.280 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.280 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.281 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.281 DEBUG:boto:Data: 2024-03-04T16:26:41.281 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.281 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.281 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.281 DEBUG:boto:Params: {} 2024-03-04T16:26:41.281 DEBUG:boto:Token: None 2024-03-04T16:26:41.281 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uDCfF+9eFLNZaiSOjbd1gyu7/5s= 2024-03-04T16:26:41.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uDCfF+9eFLNZaiSOjbd1gyu7/5s=', 'Content-Length': '0'} 2024-03-04T16:26:41.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f69e426945befe19-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.283 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.283 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.284 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.284 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.284 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.284 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.284 DEBUG:boto:Data: 2024-03-04T16:26:41.284 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.284 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.284 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.284 DEBUG:boto:Params: {} 2024-03-04T16:26:41.284 DEBUG:boto:Token: None 2024-03-04T16:26:41.284 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.285 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uDCfF+9eFLNZaiSOjbd1gyu7/5s= 2024-03-04T16:26:41.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uDCfF+9eFLNZaiSOjbd1gyu7/5s=', 'Content-Length': '0'} 2024-03-04T16:26:41.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001711a8ba34728eb9-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:41.287 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.287 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.287 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.287 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.287 DEBUG:boto:Method: GET 2024-03-04T16:26:41.287 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.287 DEBUG:boto:Data: 2024-03-04T16:26:41.288 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.288 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.288 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.288 DEBUG:boto:Params: {} 2024-03-04T16:26:41.288 DEBUG:boto:Token: None 2024-03-04T16:26:41.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da106af86f7f857f-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.332 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.332 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.332 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.332 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.332 DEBUG:boto:Method: GET 2024-03-04T16:26:41.333 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.333 DEBUG:boto:Data: 2024-03-04T16:26:41.333 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.333 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.333 DEBUG:boto:Params: {} 2024-03-04T16:26:41.333 DEBUG:boto:Token: None 2024-03-04T16:26:41.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061d07323f12e1f55-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.376 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.376 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.376 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.377 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.377 DEBUG:boto:Method: GET 2024-03-04T16:26:41.377 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.377 DEBUG:boto:Data: 2024-03-04T16:26:41.377 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.377 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.377 DEBUG:boto:Params: {} 2024-03-04T16:26:41.377 DEBUG:boto:Token: None 2024-03-04T16:26:41.377 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb7cb04f8071049-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.384 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.384 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.384 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.384 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.384 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.385 DEBUG:boto:Method: GET 2024-03-04T16:26:41.385 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.385 DEBUG:boto:Data: 2024-03-04T16:26:41.385 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.385 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.385 DEBUG:boto:Params: {} 2024-03-04T16:26:41.385 DEBUG:boto:Token: None 2024-03-04T16:26:41.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d251a39a9254c261-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.388 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.389 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.389 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.389 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.389 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.389 DEBUG:boto:Data: 2024-03-04T16:26:41.389 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.389 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.389 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.389 DEBUG:boto:Params: {} 2024-03-04T16:26:41.389 DEBUG:boto:Token: None 2024-03-04T16:26:41.390 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:16pxWVZPA6VoauS1mbTalP2YTZc= 2024-03-04T16:26:41.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:16pxWVZPA6VoauS1mbTalP2YTZc=', 'Content-Length': '0'} 2024-03-04T16:26:41.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f454c4a48124d1ce-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.392 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.392 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.392 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.392 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.393 DEBUG:boto:Data: 2024-03-04T16:26:41.393 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.393 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.393 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.393 DEBUG:boto:Params: {} 2024-03-04T16:26:41.393 DEBUG:boto:Token: None 2024-03-04T16:26:41.393 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.393 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:16pxWVZPA6VoauS1mbTalP2YTZc= 2024-03-04T16:26:41.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:16pxWVZPA6VoauS1mbTalP2YTZc=', 'Content-Length': '0'} 2024-03-04T16:26:41.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000728b2c2dd3d80d08-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:41.396 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.396 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.396 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.396 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.397 DEBUG:boto:Method: GET 2024-03-04T16:26:41.397 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.397 DEBUG:boto:Data: 2024-03-04T16:26:41.397 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.397 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.397 DEBUG:boto:Params: {} 2024-03-04T16:26:41.397 DEBUG:boto:Token: None 2024-03-04T16:26:41.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036f6e8d63c85cd2e-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.440 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.440 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:41.440 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.440 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.440 DEBUG:boto:Method: GET 2024-03-04T16:26:41.441 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.441 DEBUG:boto:Data: 2024-03-04T16:26:41.441 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.441 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.441 DEBUG:boto:Params: {} 2024-03-04T16:26:41.441 DEBUG:boto:Token: None 2024-03-04T16:26:41.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:41.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk= 2024-03-04T16:26:41.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u5PgIb6ueLkYyWHOUlhDpsHAvjk=', 'Content-Length': '0'} 2024-03-04T16:26:41.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab21ecc9d3fc8d1e-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.484 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.484 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.484 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.484 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.484 DEBUG:boto:Method: GET 2024-03-04T16:26:41.485 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.485 DEBUG:boto:Data: 2024-03-04T16:26:41.485 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.485 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:41.485 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.485 DEBUG:boto:Params: {} 2024-03-04T16:26:41.485 DEBUG:boto:Token: None 2024-03-04T16:26:41.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.485 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046af26a80f6a73a9-0065e5f641-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.488 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.489 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.489 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.489 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.489 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.489 DEBUG:boto:Method: GET 2024-03-04T16:26:41.489 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:41.489 DEBUG:boto:Data: 2024-03-04T16:26:41.489 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.489 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.490 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.490 DEBUG:boto:Params: {} 2024-03-04T16:26:41.490 DEBUG:boto:Token: None 2024-03-04T16:26:41.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.490 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559dc6f87c6e0bbf-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.536 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone2, test-zone1} 2024-03-04T16:26:41.537 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone3, test-zone1} 2024-03-04T16:26:41.537 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.537 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.537 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.537 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:41.537 DEBUG:boto:Data: 2024-03-04T16:26:41.538 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.538 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.538 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.538 DEBUG:boto:Params: {} 2024-03-04T16:26:41.538 DEBUG:boto:Token: None 2024-03-04T16:26:41.538 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PCvCt87D9JZSTSgksBgobYjApfQ= 2024-03-04T16:26:41.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PCvCt87D9JZSTSgksBgobYjApfQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000308c42104e5a3f5d-0065e5f641-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.541 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.542 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.542 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.542 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:41.542 DEBUG:boto:Data: 2024-03-04T16:26:41.542 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.542 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.542 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.542 DEBUG:boto:Params: {} 2024-03-04T16:26:41.542 DEBUG:boto:Token: None 2024-03-04T16:26:41.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PCvCt87D9JZSTSgksBgobYjApfQ= 2024-03-04T16:26:41.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PCvCt87D9JZSTSgksBgobYjApfQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5830e3be82e4d40-0065e5f641-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:41.545 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.545 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.545 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.546 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.546 DEBUG:boto:Method: GET 2024-03-04T16:26:41.546 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.546 DEBUG:boto:Data: 2024-03-04T16:26:41.546 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.546 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.546 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.546 DEBUG:boto:Params: {} 2024-03-04T16:26:41.546 DEBUG:boto:Token: None 2024-03-04T16:26:41.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/?versions 2024-03-04T16:26:41.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo= 2024-03-04T16:26:41.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo=', 'Content-Length': '0'} 2024-03-04T16:26:41.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eca476869058c20-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.549 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.550 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:41.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:41.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:41.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:41.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:41.550 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.550 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.551 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.551 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.551 DEBUG:boto:Method: GET 2024-03-04T16:26:41.551 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.551 DEBUG:boto:Data: 2024-03-04T16:26:41.551 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.551 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.551 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.551 DEBUG:boto:Params: {} 2024-03-04T16:26:41.552 DEBUG:boto:Token: None 2024-03-04T16:26:41.552 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/?versions 2024-03-04T16:26:41.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo= 2024-03-04T16:26:41.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo=', 'Content-Length': '0'} 2024-03-04T16:26:41.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325224f794aa0cf4-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.555 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.555 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:41.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:41.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:41.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:41.556 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.556 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.556 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.556 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.556 DEBUG:boto:Method: GET 2024-03-04T16:26:41.556 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.556 DEBUG:boto:Data: 2024-03-04T16:26:41.557 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.557 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.557 DEBUG:boto:Params: {} 2024-03-04T16:26:41.557 DEBUG:boto:Token: None 2024-03-04T16:26:41.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/?versions 2024-03-04T16:26:41.557 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo= 2024-03-04T16:26:41.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo=', 'Content-Length': '0'} 2024-03-04T16:26:41.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c945b515d9d3b7a7-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.560 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.561 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.561 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.561 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.561 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.561 DEBUG:boto:Method: GET 2024-03-04T16:26:41.561 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:41.561 DEBUG:boto:Data: 2024-03-04T16:26:41.561 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.562 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.562 DEBUG:boto:Params: {} 2024-03-04T16:26:41.562 DEBUG:boto:Token: None 2024-03-04T16:26:41.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/?versions 2024-03-04T16:26:41.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo= 2024-03-04T16:26:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:okiBqjH2wtiqLVLzDwCmf4pMMYo=', 'Content-Length': '0'} 2024-03-04T16:26:41.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad194087f5687b5-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.565 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:41.566 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.566 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.566 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.566 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.566 DEBUG:boto:Method: GET 2024-03-04T16:26:41.566 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.567 DEBUG:boto:Data: 2024-03-04T16:26:41.567 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.567 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.567 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.567 DEBUG:boto:Params: {} 2024-03-04T16:26:41.567 DEBUG:boto:Token: None 2024-03-04T16:26:41.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005968759e4c637d9f-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.569 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.570 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.570 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.570 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.570 DEBUG:boto:Method: GET 2024-03-04T16:26:41.570 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.570 DEBUG:boto:Data: 2024-03-04T16:26:41.570 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.570 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.570 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.570 DEBUG:boto:Params: {} 2024-03-04T16:26:41.570 DEBUG:boto:Token: None 2024-03-04T16:26:41.571 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c5860c9cb146ba6-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.572 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.573 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.573 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.573 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.573 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.573 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.573 DEBUG:boto:Data: 2024-03-04T16:26:41.573 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.573 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.573 DEBUG:boto:Params: {} 2024-03-04T16:26:41.573 DEBUG:boto:Token: None 2024-03-04T16:26:41.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ovr75FZ9WpEbH6O6lhd0CzIGNOg= 2024-03-04T16:26:41.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ovr75FZ9WpEbH6O6lhd0CzIGNOg=', 'Content-Length': '0'} 2024-03-04T16:26:41.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e8c5d3a284949fd-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.576 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.576 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.576 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.576 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.576 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.576 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.576 DEBUG:boto:Data: 2024-03-04T16:26:41.576 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.576 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.576 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.576 DEBUG:boto:Params: {} 2024-03-04T16:26:41.577 DEBUG:boto:Token: None 2024-03-04T16:26:41.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ovr75FZ9WpEbH6O6lhd0CzIGNOg= 2024-03-04T16:26:41.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ovr75FZ9WpEbH6O6lhd0CzIGNOg=', 'Content-Length': '0'} 2024-03-04T16:26:41.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d550244f2bc298c6-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:41.579 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.579 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.579 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.579 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.579 DEBUG:boto:Method: GET 2024-03-04T16:26:41.579 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.579 DEBUG:boto:Data: 2024-03-04T16:26:41.580 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.580 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.580 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.580 DEBUG:boto:Params: {} 2024-03-04T16:26:41.580 DEBUG:boto:Token: None 2024-03-04T16:26:41.580 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.580 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa3cb29164efe55f-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.583 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.583 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.583 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.583 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.583 DEBUG:boto:Method: GET 2024-03-04T16:26:41.583 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.583 DEBUG:boto:Data: 2024-03-04T16:26:41.583 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.583 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.584 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.584 DEBUG:boto:Params: {} 2024-03-04T16:26:41.584 DEBUG:boto:Token: None 2024-03-04T16:26:41.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000510da40ae8e465ea-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.586 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.586 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.586 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.587 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.587 DEBUG:boto:Method: GET 2024-03-04T16:26:41.587 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.587 DEBUG:boto:Data: 2024-03-04T16:26:41.587 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.587 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.587 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.587 DEBUG:boto:Params: {} 2024-03-04T16:26:41.587 DEBUG:boto:Token: None 2024-03-04T16:26:41.587 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.588 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd4c2ee434074ae9-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.632 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.633 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.633 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.633 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.633 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.633 DEBUG:boto:Method: GET 2024-03-04T16:26:41.633 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.633 DEBUG:boto:Data: 2024-03-04T16:26:41.633 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.634 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.634 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.634 DEBUG:boto:Params: {} 2024-03-04T16:26:41.634 DEBUG:boto:Token: None 2024-03-04T16:26:41.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6acddb6c3708b0f-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.637 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.637 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.638 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.638 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.638 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:41.638 DEBUG:boto:Data: 2024-03-04T16:26:41.638 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.638 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.638 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.638 DEBUG:boto:Params: {} 2024-03-04T16:26:41.638 DEBUG:boto:Token: None 2024-03-04T16:26:41.638 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy 2024-03-04T16:26:41.639 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3S0qAGtaeOEnUGmX16WY4K52WEY= 2024-03-04T16:26:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3S0qAGtaeOEnUGmX16WY4K52WEY=', 'Content-Length': '0'} 2024-03-04T16:26:41.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d05f99b4274cf041-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.641 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.641 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.641 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.641 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:41.641 DEBUG:boto:Data: 2024-03-04T16:26:41.642 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.642 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.642 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.642 DEBUG:boto:Params: {} 2024-03-04T16:26:41.642 DEBUG:boto:Token: None 2024-03-04T16:26:41.642 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy 2024-03-04T16:26:41.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3S0qAGtaeOEnUGmX16WY4K52WEY= 2024-03-04T16:26:41.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3S0qAGtaeOEnUGmX16WY4K52WEY=', 'Content-Length': '0'} 2024-03-04T16:26:41.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c9eea9997fe041e-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:41.645 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.645 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.645 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.645 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.645 DEBUG:boto:Method: GET 2024-03-04T16:26:41.645 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.645 DEBUG:boto:Data: 2024-03-04T16:26:41.645 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.645 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.645 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.646 DEBUG:boto:Params: {} 2024-03-04T16:26:41.646 DEBUG:boto:Token: None 2024-03-04T16:26:41.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ace61cf3bb57f91-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.648 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:41.648 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:41.648 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.648 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.648 DEBUG:boto:Method: GET 2024-03-04T16:26:41.648 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.649 DEBUG:boto:Data: 2024-03-04T16:26:41.649 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.649 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.649 DEBUG:boto:Params: {} 2024-03-04T16:26:41.649 DEBUG:boto:Token: None 2024-03-04T16:26:41.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:41.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA= 2024-03-04T16:26:41.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wus6CMfQyAEaFt58XACGJxyysVA=', 'Content-Length': '0'} 2024-03-04T16:26:41.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057b806d006afc8b2-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.651 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.651 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.651 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.651 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.652 DEBUG:boto:Method: GET 2024-03-04T16:26:41.652 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.652 DEBUG:boto:Data: 2024-03-04T16:26:41.652 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.652 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.652 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.652 DEBUG:boto:Params: {} 2024-03-04T16:26:41.652 DEBUG:boto:Token: None 2024-03-04T16:26:41.652 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.652 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba255648ea08123-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.655 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.655 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.656 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.656 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.656 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.656 DEBUG:boto:Method: GET 2024-03-04T16:26:41.656 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:41.656 DEBUG:boto:Data: 2024-03-04T16:26:41.656 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.656 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.656 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.656 DEBUG:boto:Params: {} 2024-03-04T16:26:41.656 DEBUG:boto:Token: None 2024-03-04T16:26:41.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.657 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee7e418f8c3185a-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.659 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:41.660 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.660 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.660 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.660 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.660 DEBUG:boto:Method: GET 2024-03-04T16:26:41.660 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.660 DEBUG:boto:Data: 2024-03-04T16:26:41.660 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.661 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.661 DEBUG:boto:Params: {} 2024-03-04T16:26:41.661 DEBUG:boto:Token: None 2024-03-04T16:26:41.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.661 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001518a995b7e91ea6-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.704 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.704 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.704 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.705 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.705 DEBUG:boto:Method: GET 2024-03-04T16:26:41.705 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.705 DEBUG:boto:Data: 2024-03-04T16:26:41.705 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.705 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.705 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.705 DEBUG:boto:Params: {} 2024-03-04T16:26:41.705 DEBUG:boto:Token: None 2024-03-04T16:26:41.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670bfa74eaa1f6c3-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.708 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.708 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.709 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.709 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.709 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.709 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.709 DEBUG:boto:Data: 2024-03-04T16:26:41.709 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.709 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.709 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.710 DEBUG:boto:Params: {} 2024-03-04T16:26:41.710 DEBUG:boto:Token: None 2024-03-04T16:26:41.710 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cgDAhfcEmgmc+twNblpTKGsGdXo= 2024-03-04T16:26:41.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cgDAhfcEmgmc+twNblpTKGsGdXo=', 'Content-Length': '0'} 2024-03-04T16:26:41.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdee4771d573d38e-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.712 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.713 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.713 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.713 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.713 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.713 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.713 DEBUG:boto:Data: 2024-03-04T16:26:41.713 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.714 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.714 DEBUG:boto:Params: {} 2024-03-04T16:26:41.714 DEBUG:boto:Token: None 2024-03-04T16:26:41.714 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cgDAhfcEmgmc+twNblpTKGsGdXo= 2024-03-04T16:26:41.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cgDAhfcEmgmc+twNblpTKGsGdXo=', 'Content-Length': '0'} 2024-03-04T16:26:41.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a719df628acd778-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:41.717 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.717 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.717 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.717 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.717 DEBUG:boto:Method: GET 2024-03-04T16:26:41.717 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.717 DEBUG:boto:Data: 2024-03-04T16:26:41.717 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.718 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.718 DEBUG:boto:Params: {} 2024-03-04T16:26:41.718 DEBUG:boto:Token: None 2024-03-04T16:26:41.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a32164793e259de-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.764 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.764 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.765 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.765 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.765 DEBUG:boto:Method: GET 2024-03-04T16:26:41.765 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.765 DEBUG:boto:Data: 2024-03-04T16:26:41.765 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.765 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.765 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.766 DEBUG:boto:Params: {} 2024-03-04T16:26:41.766 DEBUG:boto:Token: None 2024-03-04T16:26:41.766 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040f3b4447bb230df-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.769 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.769 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.769 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.769 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.769 DEBUG:boto:Method: GET 2024-03-04T16:26:41.769 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.769 DEBUG:boto:Data: 2024-03-04T16:26:41.770 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.770 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.770 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.770 DEBUG:boto:Params: {} 2024-03-04T16:26:41.770 DEBUG:boto:Token: None 2024-03-04T16:26:41.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.771 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008234434461898ec0-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.774 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.774 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.775 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.775 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.775 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.775 DEBUG:boto:Method: GET 2024-03-04T16:26:41.775 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.775 DEBUG:boto:Data: 2024-03-04T16:26:41.775 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.775 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.776 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.776 DEBUG:boto:Params: {} 2024-03-04T16:26:41.776 DEBUG:boto:Token: None 2024-03-04T16:26:41.776 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.776 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfad2857a3dfa1f-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.779 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.779 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.780 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.780 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.780 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.780 DEBUG:boto:Data: 2024-03-04T16:26:41.780 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.780 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.780 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.781 DEBUG:boto:Params: {} 2024-03-04T16:26:41.781 DEBUG:boto:Token: None 2024-03-04T16:26:41.781 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hnl0SaqF3oRTRXya0W93rE3g64M= 2024-03-04T16:26:41.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hnl0SaqF3oRTRXya0W93rE3g64M=', 'Content-Length': '0'} 2024-03-04T16:26:41.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eded80a4cf7f64e2-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.784 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.784 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.784 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.785 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.785 DEBUG:boto:Data: 2024-03-04T16:26:41.785 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.785 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.785 DEBUG:boto:Params: {} 2024-03-04T16:26:41.785 DEBUG:boto:Token: None 2024-03-04T16:26:41.785 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hnl0SaqF3oRTRXya0W93rE3g64M= 2024-03-04T16:26:41.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hnl0SaqF3oRTRXya0W93rE3g64M=', 'Content-Length': '0'} 2024-03-04T16:26:41.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009eb8ceb0f332c083-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:41.788 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.788 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.788 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.788 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.788 DEBUG:boto:Method: GET 2024-03-04T16:26:41.789 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.789 DEBUG:boto:Data: 2024-03-04T16:26:41.789 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.789 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.789 DEBUG:boto:Params: {} 2024-03-04T16:26:41.789 DEBUG:boto:Token: None 2024-03-04T16:26:41.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe0ec6727fd5cb54-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.832 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.832 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:41.833 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.833 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.833 DEBUG:boto:Method: GET 2024-03-04T16:26:41.833 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.833 DEBUG:boto:Data: 2024-03-04T16:26:41.833 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.833 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.833 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.834 DEBUG:boto:Params: {} 2024-03-04T16:26:41.834 DEBUG:boto:Token: None 2024-03-04T16:26:41.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:41.834 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c= 2024-03-04T16:26:41.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vme/3MEnQDV6wQMbobLLwhSON3c=', 'Content-Length': '0'} 2024-03-04T16:26:41.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d700ac557e64fe3d-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.836 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.837 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.837 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.837 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.837 DEBUG:boto:Method: GET 2024-03-04T16:26:41.837 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.837 DEBUG:boto:Data: 2024-03-04T16:26:41.837 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.837 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.838 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.838 DEBUG:boto:Params: {} 2024-03-04T16:26:41.838 DEBUG:boto:Token: None 2024-03-04T16:26:41.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba88f00dfe6f5a92-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.841 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.841 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:41.841 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:41.841 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.842 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.842 DEBUG:boto:Method: GET 2024-03-04T16:26:41.842 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:41.842 DEBUG:boto:Data: 2024-03-04T16:26:41.842 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.842 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.842 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.842 DEBUG:boto:Params: {} 2024-03-04T16:26:41.842 DEBUG:boto:Token: None 2024-03-04T16:26:41.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/ 2024-03-04T16:26:41.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ= 2024-03-04T16:26:41.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UDZIVyeht0BY/owG4U7aL1Q8DDQ=', 'Content-Length': '0'} 2024-03-04T16:26:41.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb20cca9c06126a-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.845 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:41.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:41.846 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.846 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.846 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.846 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.846 DEBUG:boto:Method: GET 2024-03-04T16:26:41.846 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.846 DEBUG:boto:Data: 2024-03-04T16:26:41.846 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.846 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.846 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.846 DEBUG:boto:Params: {} 2024-03-04T16:26:41.847 DEBUG:boto:Token: None 2024-03-04T16:26:41.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061210943fb4be3e1-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.892 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.892 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.893 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.893 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.893 DEBUG:boto:Method: GET 2024-03-04T16:26:41.893 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.893 DEBUG:boto:Data: 2024-03-04T16:26:41.893 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.893 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.893 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.893 DEBUG:boto:Params: {} 2024-03-04T16:26:41.893 DEBUG:boto:Token: None 2024-03-04T16:26:41.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.894 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a7e6157d45a9573-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.936 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.936 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.937 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.937 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.937 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.937 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.937 DEBUG:boto:Data: 2024-03-04T16:26:41.937 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.937 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.937 DEBUG:boto:Params: {} 2024-03-04T16:26:41.938 DEBUG:boto:Token: None 2024-03-04T16:26:41.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I= 2024-03-04T16:26:41.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I=', 'Content-Length': '0'} 2024-03-04T16:26:41.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001591b31dcac67872-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.940 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.940 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.941 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.941 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.941 DEBUG:boto:Method: HEAD 2024-03-04T16:26:41.941 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.941 DEBUG:boto:Data: 2024-03-04T16:26:41.941 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.941 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.941 DEBUG:boto:Params: {} 2024-03-04T16:26:41.941 DEBUG:boto:Token: None 2024-03-04T16:26:41.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I= 2024-03-04T16:26:41.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SUcwe+JmI/Jwh3MgLHFAQfK8k0I=', 'Content-Length': '0'} 2024-03-04T16:26:41.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003518ad06f9ad52c8-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:41.944 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.944 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.944 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.944 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.945 DEBUG:boto:Method: GET 2024-03-04T16:26:41.945 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.945 DEBUG:boto:Data: 2024-03-04T16:26:41.945 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:41.945 DEBUG:boto:Port: 8001 2024-03-04T16:26:41.945 DEBUG:boto:Params: {} 2024-03-04T16:26:41.945 DEBUG:boto:Token: None 2024-03-04T16:26:41.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000817dc9554c6501ef-0065e5f641-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:41.992 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.992 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:41.992 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.992 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.993 DEBUG:boto:Method: GET 2024-03-04T16:26:41.993 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.993 DEBUG:boto:Data: 2024-03-04T16:26:41.993 DEBUG:boto:Headers: {} 2024-03-04T16:26:41.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:41.993 DEBUG:boto:Port: 8000 2024-03-04T16:26:41.993 DEBUG:boto:Params: {} 2024-03-04T16:26:41.994 DEBUG:boto:Token: None 2024-03-04T16:26:41.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:41 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:41.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc= 2024-03-04T16:26:41.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECuQx3cbqftpUtaClekUm9foUVc=', 'Content-Length': '0'} 2024-03-04T16:26:41.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007697c94373df79e6-0065e5f641-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.036 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.036 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.036 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.036 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.037 DEBUG:boto:Method: GET 2024-03-04T16:26:42.037 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.037 DEBUG:boto:Data: 2024-03-04T16:26:42.037 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.037 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.037 DEBUG:boto:Params: {} 2024-03-04T16:26:42.037 DEBUG:boto:Token: None 2024-03-04T16:26:42.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ae56eba4e2fedb-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.041 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.041 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.041 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.041 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.041 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.041 DEBUG:boto:Method: GET 2024-03-04T16:26:42.042 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.042 DEBUG:boto:Data: 2024-03-04T16:26:42.042 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.042 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.042 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.042 DEBUG:boto:Params: {} 2024-03-04T16:26:42.042 DEBUG:boto:Token: None 2024-03-04T16:26:42.042 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.042 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab120be3af81151-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.045 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.045 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.046 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.046 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.046 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.046 DEBUG:boto:Data: 2024-03-04T16:26:42.046 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.046 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.046 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.046 DEBUG:boto:Params: {} 2024-03-04T16:26:42.046 DEBUG:boto:Token: None 2024-03-04T16:26:42.047 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.047 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QYGqULvhjfAOeXDX2E8JKUQHUzI= 2024-03-04T16:26:42.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QYGqULvhjfAOeXDX2E8JKUQHUzI=', 'Content-Length': '0'} 2024-03-04T16:26:42.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccb6d924bdfbd485-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.049 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.049 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.049 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.049 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.050 DEBUG:boto:Data: 2024-03-04T16:26:42.050 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.050 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.050 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.050 DEBUG:boto:Params: {} 2024-03-04T16:26:42.050 DEBUG:boto:Token: None 2024-03-04T16:26:42.050 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QYGqULvhjfAOeXDX2E8JKUQHUzI= 2024-03-04T16:26:42.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QYGqULvhjfAOeXDX2E8JKUQHUzI=', 'Content-Length': '0'} 2024-03-04T16:26:42.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbea07ce93a9035f-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:42.053 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.053 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.053 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.053 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.053 DEBUG:boto:Method: GET 2024-03-04T16:26:42.054 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.054 DEBUG:boto:Data: 2024-03-04T16:26:42.054 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.054 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.054 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.054 DEBUG:boto:Params: {} 2024-03-04T16:26:42.054 DEBUG:boto:Token: None 2024-03-04T16:26:42.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.054 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0= 2024-03-04T16:26:42.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0=', 'Content-Length': '0'} 2024-03-04T16:26:42.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4c5c0af12c699e9-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.100 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.100 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.100 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.101 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.101 DEBUG:boto:Method: GET 2024-03-04T16:26:42.101 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.101 DEBUG:boto:Data: 2024-03-04T16:26:42.101 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.101 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.101 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.101 DEBUG:boto:Params: {} 2024-03-04T16:26:42.101 DEBUG:boto:Token: None 2024-03-04T16:26:42.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0= 2024-03-04T16:26:42.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0=', 'Content-Length': '0'} 2024-03-04T16:26:42.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba63ffac8cbff9bd-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.144 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.144 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.144 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.145 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.145 DEBUG:boto:Method: GET 2024-03-04T16:26:42.145 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.145 DEBUG:boto:Data: 2024-03-04T16:26:42.145 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.145 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.145 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.145 DEBUG:boto:Params: {} 2024-03-04T16:26:42.145 DEBUG:boto:Token: None 2024-03-04T16:26:42.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.146 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff68570eb8b9af74-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.192 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.193 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.193 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.193 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.193 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.193 DEBUG:boto:Method: GET 2024-03-04T16:26:42.193 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:42.194 DEBUG:boto:Data: 2024-03-04T16:26:42.194 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.194 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.194 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.194 DEBUG:boto:Params: {} 2024-03-04T16:26:42.194 DEBUG:boto:Token: None 2024-03-04T16:26:42.194 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.195 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd33c69ecfa13db-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.200 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:42.201 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.201 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.201 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.201 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.201 DEBUG:boto:Method: GET 2024-03-04T16:26:42.201 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.201 DEBUG:boto:Data: 2024-03-04T16:26:42.201 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.202 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.202 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.202 DEBUG:boto:Params: {} 2024-03-04T16:26:42.202 DEBUG:boto:Token: None 2024-03-04T16:26:42.202 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001229438edd80089d-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.205 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.205 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.205 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.205 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.205 DEBUG:boto:Method: GET 2024-03-04T16:26:42.205 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.205 DEBUG:boto:Data: 2024-03-04T16:26:42.205 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.206 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.206 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.206 DEBUG:boto:Params: {} 2024-03-04T16:26:42.206 DEBUG:boto:Token: None 2024-03-04T16:26:42.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.206 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d53279be74e9669-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.252 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.252 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.252 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.253 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.253 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.253 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.253 DEBUG:boto:Data: 2024-03-04T16:26:42.253 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.253 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.253 DEBUG:boto:Params: {} 2024-03-04T16:26:42.253 DEBUG:boto:Token: None 2024-03-04T16:26:42.253 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nLehOuBdEqZ7eJ7XA4of3eONGTo= 2024-03-04T16:26:42.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nLehOuBdEqZ7eJ7XA4of3eONGTo=', 'Content-Length': '0'} 2024-03-04T16:26:42.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089c2b571f52421cd-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.256 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.256 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.256 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.256 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.256 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.256 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.256 DEBUG:boto:Data: 2024-03-04T16:26:42.257 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.257 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.257 DEBUG:boto:Params: {} 2024-03-04T16:26:42.257 DEBUG:boto:Token: None 2024-03-04T16:26:42.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nLehOuBdEqZ7eJ7XA4of3eONGTo= 2024-03-04T16:26:42.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nLehOuBdEqZ7eJ7XA4of3eONGTo=', 'Content-Length': '0'} 2024-03-04T16:26:42.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e748dbdea3962930-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:42.260 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.260 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.260 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.260 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.260 DEBUG:boto:Method: GET 2024-03-04T16:26:42.260 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.260 DEBUG:boto:Data: 2024-03-04T16:26:42.261 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.261 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.261 DEBUG:boto:Params: {} 2024-03-04T16:26:42.261 DEBUG:boto:Token: None 2024-03-04T16:26:42.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac021d10cd7e95e4-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.264 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.264 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.264 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.264 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.264 DEBUG:boto:Method: GET 2024-03-04T16:26:42.264 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.264 DEBUG:boto:Data: 2024-03-04T16:26:42.264 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.264 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.265 DEBUG:boto:Params: {} 2024-03-04T16:26:42.265 DEBUG:boto:Token: None 2024-03-04T16:26:42.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.265 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0099d02ae236229-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.308 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.308 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.309 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.309 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.309 DEBUG:boto:Method: GET 2024-03-04T16:26:42.309 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.309 DEBUG:boto:Data: 2024-03-04T16:26:42.309 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.309 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.309 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.310 DEBUG:boto:Params: {} 2024-03-04T16:26:42.310 DEBUG:boto:Token: None 2024-03-04T16:26:42.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000930f451d52ca756a-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.314 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.314 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.314 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.314 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.314 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.314 DEBUG:boto:Method: GET 2024-03-04T16:26:42.314 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.315 DEBUG:boto:Data: 2024-03-04T16:26:42.315 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.315 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.315 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.315 DEBUG:boto:Params: {} 2024-03-04T16:26:42.315 DEBUG:boto:Token: None 2024-03-04T16:26:42.315 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ed73aad2d3e3059-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.360 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.361 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.361 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.361 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.361 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.361 DEBUG:boto:Data: 2024-03-04T16:26:42.361 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.361 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.361 DEBUG:boto:Params: {} 2024-03-04T16:26:42.361 DEBUG:boto:Token: None 2024-03-04T16:26:42.362 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:49GcEps9XKVjkto99o9YtgXmqHc= 2024-03-04T16:26:42.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:49GcEps9XKVjkto99o9YtgXmqHc=', 'Content-Length': '0'} 2024-03-04T16:26:42.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000692a010264aa4656-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.365 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.365 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.365 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.365 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.365 DEBUG:boto:Data: 2024-03-04T16:26:42.365 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.365 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.365 DEBUG:boto:Params: {} 2024-03-04T16:26:42.365 DEBUG:boto:Token: None 2024-03-04T16:26:42.366 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:49GcEps9XKVjkto99o9YtgXmqHc= 2024-03-04T16:26:42.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:49GcEps9XKVjkto99o9YtgXmqHc=', 'Content-Length': '0'} 2024-03-04T16:26:42.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0fe12318cc34922-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:42.368 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.368 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.369 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.369 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.369 DEBUG:boto:Method: GET 2024-03-04T16:26:42.369 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.369 DEBUG:boto:Data: 2024-03-04T16:26:42.369 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.369 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.369 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.369 DEBUG:boto:Params: {} 2024-03-04T16:26:42.369 DEBUG:boto:Token: None 2024-03-04T16:26:42.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a0ce19f9f2e749f-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.372 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.372 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:42.372 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.372 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.372 DEBUG:boto:Method: GET 2024-03-04T16:26:42.372 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.372 DEBUG:boto:Data: 2024-03-04T16:26:42.372 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.372 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.373 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.373 DEBUG:boto:Params: {} 2024-03-04T16:26:42.373 DEBUG:boto:Token: None 2024-03-04T16:26:42.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:42.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio= 2024-03-04T16:26:42.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9x07Tw0voCKWkVulHAMf8Ie8Sio=', 'Content-Length': '0'} 2024-03-04T16:26:42.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006770dd1627c3dcb4-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.416 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.416 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.416 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.417 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.417 DEBUG:boto:Method: GET 2024-03-04T16:26:42.417 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.417 DEBUG:boto:Data: 2024-03-04T16:26:42.417 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.417 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.417 DEBUG:boto:Params: {} 2024-03-04T16:26:42.417 DEBUG:boto:Token: None 2024-03-04T16:26:42.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c4a4f3594256dd-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.421 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.421 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.421 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.421 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.421 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.422 DEBUG:boto:Method: GET 2024-03-04T16:26:42.422 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:42.422 DEBUG:boto:Data: 2024-03-04T16:26:42.422 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.422 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:26:42.422 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.422 DEBUG:boto:Params: {} 2024-03-04T16:26:42.422 DEBUG:boto:Token: None 2024-03-04T16:26:42.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157788995f74608a-0065e5f642-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.468 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.469 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone3, test-zone1} 2024-03-04T16:26:42.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone3, test-zone2} 2024-03-04T16:26:42.469 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.469 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.469 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.469 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:42.469 DEBUG:boto:Data: 2024-03-04T16:26:42.469 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.469 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.470 DEBUG:boto:Params: {} 2024-03-04T16:26:42.470 DEBUG:boto:Token: None 2024-03-04T16:26:42.470 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hwMMCaENecNu6A09EECXwcdOWpU= 2024-03-04T16:26:42.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hwMMCaENecNu6A09EECXwcdOWpU=', 'Content-Length': '0'} 2024-03-04T16:26:42.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da44dcae54e2034-0065e5f642-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.473 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.474 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.474 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.474 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:26:42.474 DEBUG:boto:Data: 2024-03-04T16:26:42.474 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.474 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.475 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.475 DEBUG:boto:Params: {} 2024-03-04T16:26:42.475 DEBUG:boto:Token: None 2024-03-04T16:26:42.475 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.475 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hwMMCaENecNu6A09EECXwcdOWpU= 2024-03-04T16:26:42.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hwMMCaENecNu6A09EECXwcdOWpU=', 'Content-Length': '0'} 2024-03-04T16:26:42.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7a07da974b57d29-0065e5f642-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:26:42.479 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.479 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.479 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.479 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.480 DEBUG:boto:Method: GET 2024-03-04T16:26:42.480 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.480 DEBUG:boto:Data: 2024-03-04T16:26:42.480 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.480 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.480 DEBUG:boto:Params: {} 2024-03-04T16:26:42.480 DEBUG:boto:Token: None 2024-03-04T16:26:42.480 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/?versions 2024-03-04T16:26:42.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw= 2024-03-04T16:26:42.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw=', 'Content-Length': '0'} 2024-03-04T16:26:42.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b5ef041be64ffde-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.483 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.484 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:42.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:42.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:42.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:42.484 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:26:42.484 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.485 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.485 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.485 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.485 DEBUG:boto:Method: GET 2024-03-04T16:26:42.485 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.485 DEBUG:boto:Data: 2024-03-04T16:26:42.485 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.485 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.485 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.485 DEBUG:boto:Params: {} 2024-03-04T16:26:42.485 DEBUG:boto:Token: None 2024-03-04T16:26:42.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/?versions 2024-03-04T16:26:42.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw= 2024-03-04T16:26:42.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw=', 'Content-Length': '0'} 2024-03-04T16:26:42.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889cffba5440716f-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.489 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.489 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:26:42.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:26:42.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:26:42.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:26:42.489 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.489 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.490 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.490 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.490 DEBUG:boto:Method: GET 2024-03-04T16:26:42.490 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.490 DEBUG:boto:Data: 2024-03-04T16:26:42.490 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.490 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.490 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.490 DEBUG:boto:Params: {} 2024-03-04T16:26:42.490 DEBUG:boto:Token: None 2024-03-04T16:26:42.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/?versions 2024-03-04T16:26:42.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw= 2024-03-04T16:26:42.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw=', 'Content-Length': '0'} 2024-03-04T16:26:42.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006632a601f598b3a6-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.494 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.494 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.494 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.494 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.495 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.495 DEBUG:boto:Method: GET 2024-03-04T16:26:42.495 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:26:42.495 DEBUG:boto:Data: 2024-03-04T16:26:42.495 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.495 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.495 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.495 DEBUG:boto:Params: {} 2024-03-04T16:26:42.495 DEBUG:boto:Token: None 2024-03-04T16:26:42.495 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/?versions 2024-03-04T16:26:42.495 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw= 2024-03-04T16:26:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+q/KHN3eHxfPtbpYnLVhsH5z8xw=', 'Content-Length': '0'} 2024-03-04T16:26:42.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f1424f54abdb796-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.498 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:42.499 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.499 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.499 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.499 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.499 DEBUG:boto:Method: GET 2024-03-04T16:26:42.499 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.500 DEBUG:boto:Data: 2024-03-04T16:26:42.500 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.500 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.500 DEBUG:boto:Params: {} 2024-03-04T16:26:42.500 DEBUG:boto:Token: None 2024-03-04T16:26:42.500 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.500 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8319fa4830d135b-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.502 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.503 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.503 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.503 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.503 DEBUG:boto:Method: GET 2024-03-04T16:26:42.503 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.503 DEBUG:boto:Data: 2024-03-04T16:26:42.503 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.503 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.503 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.503 DEBUG:boto:Params: {} 2024-03-04T16:26:42.503 DEBUG:boto:Token: None 2024-03-04T16:26:42.504 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.504 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bd5b1e845d07971-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.506 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.506 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.506 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.506 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.506 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.506 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.506 DEBUG:boto:Data: 2024-03-04T16:26:42.506 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.506 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.507 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.507 DEBUG:boto:Params: {} 2024-03-04T16:26:42.507 DEBUG:boto:Token: None 2024-03-04T16:26:42.507 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.507 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J86DlCYddLgj29oZFoHP4FzVGhU= 2024-03-04T16:26:42.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J86DlCYddLgj29oZFoHP4FzVGhU=', 'Content-Length': '0'} 2024-03-04T16:26:42.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0fb63dfecc9938a-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.509 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.509 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.509 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.509 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.509 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.509 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.509 DEBUG:boto:Data: 2024-03-04T16:26:42.510 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.510 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.510 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.510 DEBUG:boto:Params: {} 2024-03-04T16:26:42.510 DEBUG:boto:Token: None 2024-03-04T16:26:42.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J86DlCYddLgj29oZFoHP4FzVGhU= 2024-03-04T16:26:42.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J86DlCYddLgj29oZFoHP4FzVGhU=', 'Content-Length': '0'} 2024-03-04T16:26:42.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b9677ab58bba79c-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:42.512 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.512 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.512 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.512 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.513 DEBUG:boto:Method: GET 2024-03-04T16:26:42.513 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.513 DEBUG:boto:Data: 2024-03-04T16:26:42.513 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.513 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.513 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.513 DEBUG:boto:Params: {} 2024-03-04T16:26:42.513 DEBUG:boto:Token: None 2024-03-04T16:26:42.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad6308284bcfd43f-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.516 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.516 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.516 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.516 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.516 DEBUG:boto:Method: GET 2024-03-04T16:26:42.516 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.516 DEBUG:boto:Data: 2024-03-04T16:26:42.516 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.516 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.517 DEBUG:boto:Params: {} 2024-03-04T16:26:42.517 DEBUG:boto:Token: None 2024-03-04T16:26:42.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d71ff56d89eba98-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.519 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.519 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.519 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.519 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.519 DEBUG:boto:Method: GET 2024-03-04T16:26:42.519 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.519 DEBUG:boto:Data: 2024-03-04T16:26:42.520 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.520 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.520 DEBUG:boto:Params: {} 2024-03-04T16:26:42.520 DEBUG:boto:Token: None 2024-03-04T16:26:42.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfa2edd284cf2de0-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.564 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.565 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.565 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.565 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.565 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.566 DEBUG:boto:Method: GET 2024-03-04T16:26:42.566 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.566 DEBUG:boto:Data: 2024-03-04T16:26:42.566 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.566 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.566 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.566 DEBUG:boto:Params: {} 2024-03-04T16:26:42.566 DEBUG:boto:Token: None 2024-03-04T16:26:42.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008404c5991a80bc2f-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.569 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.570 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.570 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.570 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.570 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:42.570 DEBUG:boto:Data: 2024-03-04T16:26:42.570 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.570 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.570 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.570 DEBUG:boto:Params: {} 2024-03-04T16:26:42.571 DEBUG:boto:Token: None 2024-03-04T16:26:42.571 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy 2024-03-04T16:26:42.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YwKvM6YgxM5deY+LGwIECzB2ILM= 2024-03-04T16:26:42.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YwKvM6YgxM5deY+LGwIECzB2ILM=', 'Content-Length': '0'} 2024-03-04T16:26:42.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae9b87760f494a66-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.574 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.574 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.574 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.574 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:26:42.574 DEBUG:boto:Data: 2024-03-04T16:26:42.574 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.574 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.574 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.574 DEBUG:boto:Params: {} 2024-03-04T16:26:42.575 DEBUG:boto:Token: None 2024-03-04T16:26:42.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy 2024-03-04T16:26:42.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YwKvM6YgxM5deY+LGwIECzB2ILM= 2024-03-04T16:26:42.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YwKvM6YgxM5deY+LGwIECzB2ILM=', 'Content-Length': '0'} 2024-03-04T16:26:42.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c0d5d84d2b86575-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:26:42.577 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.577 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.577 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.578 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.578 DEBUG:boto:Method: GET 2024-03-04T16:26:42.578 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.578 DEBUG:boto:Data: 2024-03-04T16:26:42.578 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.578 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.578 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.578 DEBUG:boto:Params: {} 2024-03-04T16:26:42.578 DEBUG:boto:Token: None 2024-03-04T16:26:42.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.579 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db284092f2b154c7-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.581 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:26:42.581 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:26:42.581 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.581 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.581 DEBUG:boto:Method: GET 2024-03-04T16:26:42.581 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.581 DEBUG:boto:Data: 2024-03-04T16:26:42.581 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.582 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.582 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.582 DEBUG:boto:Params: {} 2024-03-04T16:26:42.582 DEBUG:boto:Token: None 2024-03-04T16:26:42.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:26:42.582 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg= 2024-03-04T16:26:42.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hDA23/LVz71XSzMHbva4TttpTdg=', 'Content-Length': '0'} 2024-03-04T16:26:42.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000565bfb6d0840d3af-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.585 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.585 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.585 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.585 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.585 DEBUG:boto:Method: GET 2024-03-04T16:26:42.585 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.585 DEBUG:boto:Data: 2024-03-04T16:26:42.585 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.585 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.585 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.585 DEBUG:boto:Params: {} 2024-03-04T16:26:42.585 DEBUG:boto:Token: None 2024-03-04T16:26:42.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a2d0087f84911fe-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.632 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.633 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.633 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.633 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.633 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.633 DEBUG:boto:Method: GET 2024-03-04T16:26:42.634 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:26:42.634 DEBUG:boto:Data: 2024-03-04T16:26:42.634 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.634 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.634 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.634 DEBUG:boto:Params: {} 2024-03-04T16:26:42.634 DEBUG:boto:Token: None 2024-03-04T16:26:42.635 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.635 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc086d2bca5032d8-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.638 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:42.638 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.638 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.638 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.638 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.638 DEBUG:boto:Method: GET 2024-03-04T16:26:42.639 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.639 DEBUG:boto:Data: 2024-03-04T16:26:42.639 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.639 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.639 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.639 DEBUG:boto:Params: {} 2024-03-04T16:26:42.639 DEBUG:boto:Token: None 2024-03-04T16:26:42.639 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.639 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710a0e6dd24c31fd-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.684 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.684 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.684 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.684 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.685 DEBUG:boto:Method: GET 2024-03-04T16:26:42.685 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.685 DEBUG:boto:Data: 2024-03-04T16:26:42.685 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.686 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.686 DEBUG:boto:Params: {} 2024-03-04T16:26:42.686 DEBUG:boto:Token: None 2024-03-04T16:26:42.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.687 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5bc0fc3847104bc-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.689 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.689 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.689 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.690 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.690 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.690 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.690 DEBUG:boto:Data: 2024-03-04T16:26:42.690 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.690 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.690 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.690 DEBUG:boto:Params: {} 2024-03-04T16:26:42.690 DEBUG:boto:Token: None 2024-03-04T16:26:42.690 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j0+eD8XpOp6liiqZ48w+c/temRc= 2024-03-04T16:26:42.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j0+eD8XpOp6liiqZ48w+c/temRc=', 'Content-Length': '0'} 2024-03-04T16:26:42.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070b2f3d82683c047-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.693 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.693 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.693 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.693 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.693 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.693 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.693 DEBUG:boto:Data: 2024-03-04T16:26:42.693 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.694 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.694 DEBUG:boto:Params: {} 2024-03-04T16:26:42.694 DEBUG:boto:Token: None 2024-03-04T16:26:42.694 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j0+eD8XpOp6liiqZ48w+c/temRc= 2024-03-04T16:26:42.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j0+eD8XpOp6liiqZ48w+c/temRc=', 'Content-Length': '0'} 2024-03-04T16:26:42.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d40eee0a7023988a-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:42.696 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.696 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.697 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.697 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.697 DEBUG:boto:Method: GET 2024-03-04T16:26:42.697 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.697 DEBUG:boto:Data: 2024-03-04T16:26:42.697 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.698 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.698 DEBUG:boto:Params: {} 2024-03-04T16:26:42.698 DEBUG:boto:Token: None 2024-03-04T16:26:42.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f9e5777aac497c-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.744 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.744 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.744 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.744 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.745 DEBUG:boto:Method: GET 2024-03-04T16:26:42.745 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.745 DEBUG:boto:Data: 2024-03-04T16:26:42.745 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.745 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.745 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.745 DEBUG:boto:Params: {} 2024-03-04T16:26:42.745 DEBUG:boto:Token: None 2024-03-04T16:26:42.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d178540a9fcd11e2-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.792 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.792 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.793 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.793 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.793 DEBUG:boto:Method: GET 2024-03-04T16:26:42.793 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.793 DEBUG:boto:Data: 2024-03-04T16:26:42.793 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.793 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.793 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.793 DEBUG:boto:Params: {} 2024-03-04T16:26:42.794 DEBUG:boto:Token: None 2024-03-04T16:26:42.794 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d0362bb1e94596-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.797 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.797 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.797 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.798 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.798 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.798 DEBUG:boto:Method: GET 2024-03-04T16:26:42.798 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.798 DEBUG:boto:Data: 2024-03-04T16:26:42.798 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.798 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.798 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.798 DEBUG:boto:Params: {} 2024-03-04T16:26:42.798 DEBUG:boto:Token: None 2024-03-04T16:26:42.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067eef616d30258da-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.802 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.802 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.802 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.802 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.802 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.802 DEBUG:boto:Data: 2024-03-04T16:26:42.802 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.802 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.803 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.803 DEBUG:boto:Params: {} 2024-03-04T16:26:42.803 DEBUG:boto:Token: None 2024-03-04T16:26:42.803 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.803 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AgTd8nvhPbv3bqUjhIaAPHc0cSg= 2024-03-04T16:26:42.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AgTd8nvhPbv3bqUjhIaAPHc0cSg=', 'Content-Length': '0'} 2024-03-04T16:26:42.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d032d2843088fae-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.806 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.806 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.806 DEBUG:boto:Method: HEAD 2024-03-04T16:26:42.806 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.806 DEBUG:boto:Data: 2024-03-04T16:26:42.806 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.806 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.807 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.807 DEBUG:boto:Params: {} 2024-03-04T16:26:42.807 DEBUG:boto:Token: None 2024-03-04T16:26:42.807 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.807 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AgTd8nvhPbv3bqUjhIaAPHc0cSg= 2024-03-04T16:26:42.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AgTd8nvhPbv3bqUjhIaAPHc0cSg=', 'Content-Length': '0'} 2024-03-04T16:26:42.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af04ef629def5aaf-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:26:42.810 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.810 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.810 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.810 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.810 DEBUG:boto:Method: GET 2024-03-04T16:26:42.810 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.810 DEBUG:boto:Data: 2024-03-04T16:26:42.810 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.810 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.810 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.810 DEBUG:boto:Params: {} 2024-03-04T16:26:42.811 DEBUG:boto:Token: None 2024-03-04T16:26:42.811 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.811 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff209a76ee7b633a-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.856 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.856 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:26:42.856 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.857 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.857 DEBUG:boto:Method: GET 2024-03-04T16:26:42.857 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.857 DEBUG:boto:Data: 2024-03-04T16:26:42.857 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.857 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.857 DEBUG:boto:Params: {} 2024-03-04T16:26:42.857 DEBUG:boto:Token: None 2024-03-04T16:26:42.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:26:42.858 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI= 2024-03-04T16:26:42.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qSezmUqPUcu8ego8NLYwl9Qf1GI=', 'Content-Length': '0'} 2024-03-04T16:26:42.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000878ea43d2d490738-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.904 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.904 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.904 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.905 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.905 DEBUG:boto:Method: GET 2024-03-04T16:26:42.905 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.905 DEBUG:boto:Data: 2024-03-04T16:26:42.905 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.905 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.905 DEBUG:boto:Params: {} 2024-03-04T16:26:42.905 DEBUG:boto:Token: None 2024-03-04T16:26:42.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.906 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000386a8b30d4070c54-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.952 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.953 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:42.953 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:42.953 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.953 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.953 DEBUG:boto:Method: GET 2024-03-04T16:26:42.953 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:26:42.953 DEBUG:boto:Data: 2024-03-04T16:26:42.953 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.953 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:42.953 DEBUG:boto:Port: 8000 2024-03-04T16:26:42.954 DEBUG:boto:Params: {} 2024-03-04T16:26:42.954 DEBUG:boto:Token: None 2024-03-04T16:26:42.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/ 2024-03-04T16:26:42.954 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o= 2024-03-04T16:26:42.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FB8oPF3atLMarplHPGmWOV4l01o=', 'Content-Length': '0'} 2024-03-04T16:26:42.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ba38fdeaffcdd9-0065e5f642-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:42.957 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:42.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:42.957 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.958 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:42.958 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.958 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.958 DEBUG:boto:Method: GET 2024-03-04T16:26:42.958 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.958 DEBUG:boto:Data: 2024-03-04T16:26:42.958 DEBUG:boto:Headers: {} 2024-03-04T16:26:42.958 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:42.958 DEBUG:boto:Port: 8001 2024-03-04T16:26:42.958 DEBUG:boto:Params: {} 2024-03-04T16:26:42.958 DEBUG:boto:Token: None 2024-03-04T16:26:42.959 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:42 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:42.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0= 2024-03-04T16:26:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8SlT24WktgrieHGzZ3UY2uEtpo0=', 'Content-Length': '0'} 2024-03-04T16:26:42.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecb6d28dfa9b4bfd-0065e5f642-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.008 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.008 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.008 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.008 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.009 DEBUG:boto:Method: GET 2024-03-04T16:26:43.009 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.009 DEBUG:boto:Data: 2024-03-04T16:26:43.009 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.009 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.009 DEBUG:boto:Params: {} 2024-03-04T16:26:43.009 DEBUG:boto:Token: None 2024-03-04T16:26:43.009 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.009 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY= 2024-03-04T16:26:43.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY=', 'Content-Length': '0'} 2024-03-04T16:26:43.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056462184536ddce4-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.012 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.012 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.012 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.012 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.012 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.012 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.012 DEBUG:boto:Data: 2024-03-04T16:26:43.012 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.013 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.013 DEBUG:boto:Params: {} 2024-03-04T16:26:43.013 DEBUG:boto:Token: None 2024-03-04T16:26:43.013 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kGmZDmmW4QaTegnN5YlOydbS78Y= 2024-03-04T16:26:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kGmZDmmW4QaTegnN5YlOydbS78Y=', 'Content-Length': '0'} 2024-03-04T16:26:43.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7138315e7baef56-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.015 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.015 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.015 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.015 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.016 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.016 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.016 DEBUG:boto:Data: 2024-03-04T16:26:43.016 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.016 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.016 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.016 DEBUG:boto:Params: {} 2024-03-04T16:26:43.016 DEBUG:boto:Token: None 2024-03-04T16:26:43.016 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kGmZDmmW4QaTegnN5YlOydbS78Y= 2024-03-04T16:26:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kGmZDmmW4QaTegnN5YlOydbS78Y=', 'Content-Length': '0'} 2024-03-04T16:26:43.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7f80b0c5b5e0521-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:43.019 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.019 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.019 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.019 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.019 DEBUG:boto:Method: GET 2024-03-04T16:26:43.019 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.019 DEBUG:boto:Data: 2024-03-04T16:26:43.019 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.020 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.020 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.020 DEBUG:boto:Params: {} 2024-03-04T16:26:43.020 DEBUG:boto:Token: None 2024-03-04T16:26:43.024 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.024 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY= 2024-03-04T16:26:43.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY=', 'Content-Length': '0'} 2024-03-04T16:26:43.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3412676078634c1-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.068 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.068 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.068 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.069 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.069 DEBUG:boto:Method: GET 2024-03-04T16:26:43.069 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.069 DEBUG:boto:Data: 2024-03-04T16:26:43.069 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.069 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.069 DEBUG:boto:Params: {} 2024-03-04T16:26:43.069 DEBUG:boto:Token: None 2024-03-04T16:26:43.069 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY= 2024-03-04T16:26:43.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY=', 'Content-Length': '0'} 2024-03-04T16:26:43.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a4bd235e67ecf6b-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.072 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.072 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.072 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.072 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.072 DEBUG:boto:Method: GET 2024-03-04T16:26:43.072 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.073 DEBUG:boto:Data: 2024-03-04T16:26:43.073 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.073 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.073 DEBUG:boto:Params: {} 2024-03-04T16:26:43.073 DEBUG:boto:Token: None 2024-03-04T16:26:43.073 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.073 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a8cf4972e0438c-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.120 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.121 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.121 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.121 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.121 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.121 DEBUG:boto:Method: GET 2024-03-04T16:26:43.121 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.121 DEBUG:boto:Data: 2024-03-04T16:26:43.121 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.121 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.122 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.122 DEBUG:boto:Params: {} 2024-03-04T16:26:43.122 DEBUG:boto:Token: None 2024-03-04T16:26:43.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529335f6332bd667-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.125 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.125 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.125 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.126 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.126 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.126 DEBUG:boto:Data: 2024-03-04T16:26:43.126 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.126 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.126 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.126 DEBUG:boto:Params: {} 2024-03-04T16:26:43.126 DEBUG:boto:Token: None 2024-03-04T16:26:43.127 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.127 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t2qoQjmebX1t6NwLQgM/5a4EUN0= 2024-03-04T16:26:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t2qoQjmebX1t6NwLQgM/5a4EUN0=', 'Content-Length': '0'} 2024-03-04T16:26:43.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bc0696afb90980b-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.129 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.130 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.130 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.130 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.130 DEBUG:boto:Data: 2024-03-04T16:26:43.130 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.130 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.130 DEBUG:boto:Params: {} 2024-03-04T16:26:43.130 DEBUG:boto:Token: None 2024-03-04T16:26:43.131 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.131 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t2qoQjmebX1t6NwLQgM/5a4EUN0= 2024-03-04T16:26:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t2qoQjmebX1t6NwLQgM/5a4EUN0=', 'Content-Length': '0'} 2024-03-04T16:26:43.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd2eb4f9561f6ffc-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:26:43.134 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.134 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.134 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.134 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.134 DEBUG:boto:Method: GET 2024-03-04T16:26:43.134 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.134 DEBUG:boto:Data: 2024-03-04T16:26:43.135 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.135 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.135 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.135 DEBUG:boto:Params: {} 2024-03-04T16:26:43.135 DEBUG:boto:Token: None 2024-03-04T16:26:43.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.136 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY= 2024-03-04T16:26:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY=', 'Content-Length': '0'} 2024-03-04T16:26:43.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae5f237d96ac3ad3-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.180 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:26:43.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.181 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.181 DEBUG:boto:Method: GET 2024-03-04T16:26:43.181 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.181 DEBUG:boto:Data: 2024-03-04T16:26:43.181 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.181 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.181 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.181 DEBUG:boto:Params: {} 2024-03-04T16:26:43.182 DEBUG:boto:Token: None 2024-03-04T16:26:43.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:26:43.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY= 2024-03-04T16:26:43.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ss05xUjTJ0z+pIfgVSGU0vflnsY=', 'Content-Length': '0'} 2024-03-04T16:26:43.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d764bcc5cfbb50a9-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.184 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.184 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.184 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.185 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.185 DEBUG:boto:Method: GET 2024-03-04T16:26:43.185 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.185 DEBUG:boto:Data: 2024-03-04T16:26:43.185 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.185 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.185 DEBUG:boto:Params: {} 2024-03-04T16:26:43.185 DEBUG:boto:Token: None 2024-03-04T16:26:43.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231788eb2fde7c38-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.232 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.233 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.233 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.233 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.233 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.233 DEBUG:boto:Method: GET 2024-03-04T16:26:43.233 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:26:43.233 DEBUG:boto:Data: 2024-03-04T16:26:43.233 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.234 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.234 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.234 DEBUG:boto:Params: {} 2024-03-04T16:26:43.234 DEBUG:boto:Token: None 2024-03-04T16:26:43.234 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c2e7d04ef92d71-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.237 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:43.238 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.238 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.238 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.238 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.238 DEBUG:boto:Method: GET 2024-03-04T16:26:43.238 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.238 DEBUG:boto:Data: 2024-03-04T16:26:43.238 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.238 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.238 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.238 DEBUG:boto:Params: {} 2024-03-04T16:26:43.239 DEBUG:boto:Token: None 2024-03-04T16:26:43.239 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.239 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a609ef8b148d2b0-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.241 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.241 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.241 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.241 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.241 DEBUG:boto:Method: GET 2024-03-04T16:26:43.241 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.242 DEBUG:boto:Data: 2024-03-04T16:26:43.242 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.242 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.242 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.242 DEBUG:boto:Params: {} 2024-03-04T16:26:43.242 DEBUG:boto:Token: None 2024-03-04T16:26:43.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009822370d7c11b6b-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.244 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.244 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.244 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.245 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.245 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.245 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.245 DEBUG:boto:Data: 2024-03-04T16:26:43.245 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.245 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.245 DEBUG:boto:Params: {} 2024-03-04T16:26:43.245 DEBUG:boto:Token: None 2024-03-04T16:26:43.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wmcNthpTnksBTItKjcxmjxivtdk= 2024-03-04T16:26:43.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wmcNthpTnksBTItKjcxmjxivtdk=', 'Content-Length': '0'} 2024-03-04T16:26:43.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d901b5aa9a7831dd-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.247 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.248 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.248 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.248 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.248 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.248 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.248 DEBUG:boto:Data: 2024-03-04T16:26:43.248 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.248 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.248 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.248 DEBUG:boto:Params: {} 2024-03-04T16:26:43.248 DEBUG:boto:Token: None 2024-03-04T16:26:43.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wmcNthpTnksBTItKjcxmjxivtdk= 2024-03-04T16:26:43.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wmcNthpTnksBTItKjcxmjxivtdk=', 'Content-Length': '0'} 2024-03-04T16:26:43.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007edcbd2ea09f86f5-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:43.251 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.251 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.251 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.251 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.251 DEBUG:boto:Method: GET 2024-03-04T16:26:43.251 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.251 DEBUG:boto:Data: 2024-03-04T16:26:43.251 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.251 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.252 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.252 DEBUG:boto:Params: {} 2024-03-04T16:26:43.252 DEBUG:boto:Token: None 2024-03-04T16:26:43.252 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.252 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5ff9a9688010e96-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.255 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.255 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.255 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.255 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.255 DEBUG:boto:Method: GET 2024-03-04T16:26:43.255 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.255 DEBUG:boto:Data: 2024-03-04T16:26:43.255 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.255 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.255 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.256 DEBUG:boto:Params: {} 2024-03-04T16:26:43.256 DEBUG:boto:Token: None 2024-03-04T16:26:43.256 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.256 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1c8a836d2182c92-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.304 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.304 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.305 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.305 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.305 DEBUG:boto:Method: GET 2024-03-04T16:26:43.305 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.305 DEBUG:boto:Data: 2024-03-04T16:26:43.305 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.305 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.305 DEBUG:boto:Params: {} 2024-03-04T16:26:43.305 DEBUG:boto:Token: None 2024-03-04T16:26:43.306 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003642e480ea44cb3e-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.312 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.313 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.313 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.314 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.314 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.314 DEBUG:boto:Method: GET 2024-03-04T16:26:43.314 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.314 DEBUG:boto:Data: 2024-03-04T16:26:43.315 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.315 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.315 DEBUG:boto:Params: {} 2024-03-04T16:26:43.315 DEBUG:boto:Token: None 2024-03-04T16:26:43.316 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.316 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000068fd25e7e397332-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.364 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.365 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.365 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.365 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.365 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.365 DEBUG:boto:Data: 2024-03-04T16:26:43.365 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.365 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.365 DEBUG:boto:Params: {} 2024-03-04T16:26:43.366 DEBUG:boto:Token: None 2024-03-04T16:26:43.366 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WP9TznDpfOUGpv3GkzNuX/SQJeI= 2024-03-04T16:26:43.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WP9TznDpfOUGpv3GkzNuX/SQJeI=', 'Content-Length': '0'} 2024-03-04T16:26:43.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000679e80c28fb67240-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.368 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.369 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.369 DEBUG:boto:Method: HEAD 2024-03-04T16:26:43.369 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.369 DEBUG:boto:Data: 2024-03-04T16:26:43.369 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.369 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.369 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.369 DEBUG:boto:Params: {} 2024-03-04T16:26:43.369 DEBUG:boto:Token: None 2024-03-04T16:26:43.369 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WP9TznDpfOUGpv3GkzNuX/SQJeI= 2024-03-04T16:26:43.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WP9TznDpfOUGpv3GkzNuX/SQJeI=', 'Content-Length': '0'} 2024-03-04T16:26:43.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053a7a8bff9d263e7-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:26:43.372 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.372 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.372 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.373 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.373 DEBUG:boto:Method: GET 2024-03-04T16:26:43.373 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.373 DEBUG:boto:Data: 2024-03-04T16:26:43.373 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.373 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.373 DEBUG:boto:Params: {} 2024-03-04T16:26:43.373 DEBUG:boto:Token: None 2024-03-04T16:26:43.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.374 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000863f55b8b72e8613-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.420 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.420 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:26:43.421 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.421 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.421 DEBUG:boto:Method: GET 2024-03-04T16:26:43.421 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.421 DEBUG:boto:Data: 2024-03-04T16:26:43.421 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.421 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.422 DEBUG:boto:Params: {} 2024-03-04T16:26:43.422 DEBUG:boto:Token: None 2024-03-04T16:26:43.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:26:43.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw= 2024-03-04T16:26:43.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NR3X2WxyPDeDywHdjGhawXKnVsw=', 'Content-Length': '0'} 2024-03-04T16:26:43.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049f700c346cb53d4-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.472 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.472 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.472 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.473 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.473 DEBUG:boto:Method: GET 2024-03-04T16:26:43.473 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.473 DEBUG:boto:Data: 2024-03-04T16:26:43.473 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.473 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:26:43.473 DEBUG:boto:Port: 8001 2024-03-04T16:26:43.473 DEBUG:boto:Params: {} 2024-03-04T16:26:43.473 DEBUG:boto:Token: None 2024-03-04T16:26:43.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872c1b73a4ce6014-0065e5f643-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.476 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.477 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:26:43.477 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:26:43.477 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.477 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.477 DEBUG:boto:Method: GET 2024-03-04T16:26:43.477 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:26:43.477 DEBUG:boto:Data: 2024-03-04T16:26:43.478 DEBUG:boto:Headers: {} 2024-03-04T16:26:43.478 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:26:43.478 DEBUG:boto:Port: 8000 2024-03-04T16:26:43.478 DEBUG:boto:Params: {} 2024-03-04T16:26:43.478 DEBUG:boto:Token: None 2024-03-04T16:26:43.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:26:43 GMT /sbnvju-20/ 2024-03-04T16:26:43.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM= 2024-03-04T16:26:43.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:26:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oRCwrEPjB1kqS6LdjxG6QaT2zM=', 'Content-Length': '0'} 2024-03-04T16:26:43.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054536a160f674973-0065e5f643-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:26:43.481 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:26:43.482 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone3, test-zone2} 2024-03-04T16:26:43.482 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-21 2024-03-04T16:26:43.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:43.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:26:43.483 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:26:43.543 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:26:43.543 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:26:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:43.595+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:43.596 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:43.596 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:48.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:48.597+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:48.597 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:48.598 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:53.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:53.598+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:53.600 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:53.600 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:58.600 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:26:58.600+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:26:58.600 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:26:58.600 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:03.601 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:03.601+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:03.601 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:03.602 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:08.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:08.602+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:08.603 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:08.603 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:13.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:13.604+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:13.604 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:13.604 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:18.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:18.605+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:18.606 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:18.606 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:23.607 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:23.606+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:23.607 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:23.607 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:28.608 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:28.608+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:28.609 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:28.609 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:33.610 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:33.609+0000 7fd504867800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:33.610 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000008.63.12, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:33.611 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:38.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:38.610+0000 7fd504867800 1 bucket sync caught up with source: 2024-03-04T16:27:38.611 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:38.611 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:38.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:38.610+0000 7fd504867800 0 bucket checkpoint complete 2024-03-04T16:27:38.634 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-21 2024-03-04T16:27:38.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.635 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:38.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:38.752+0000 7f74e0a9d800 1 bucket sync caught up with source: 2024-03-04T16:27:38.752 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:38.752 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000010.67.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:27:38.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:38.752+0000 7f74e0a9d800 0 bucket checkpoint complete 2024-03-04T16:27:38.771 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-21 2024-03-04T16:27:38.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.772 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:38.827 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:38.827 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:38.877+0000 7f47de28c800 1 bucket sync caught up with source: 2024-03-04T16:27:38.878 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.52.10, , 00000000010.44.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:27:38.878 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.52.10, , 00000000010.44.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:27:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:38.877+0000 7f47de28c800 0 bucket checkpoint complete 2024-03-04T16:27:38.896 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-21 2024-03-04T16:27:38.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:38.897 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:38.951 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:38.951 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:39.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:39.002+0000 7f855c467800 1 bucket sync caught up with source: 2024-03-04T16:27:39.002 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.52.10, , 00000000010.44.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:27:39.002 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.52.10, , 00000000010.44.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:27:39.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:39.002+0000 7f855c467800 0 bucket checkpoint complete 2024-03-04T16:27:39.021 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-21 2024-03-04T16:27:39.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:39.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:39.021 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:39.125+0000 7f9f2462e800 1 bucket sync caught up with source: 2024-03-04T16:27:39.126 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.53.10, , 00000000003.31.10, , 00000000015.45.10, , , , , ] 2024-03-04T16:27:39.126 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.53.10, , 00000000003.31.10, , 00000000015.45.10, , , , , ] 2024-03-04T16:27:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:39.125+0000 7f9f2462e800 0 bucket checkpoint complete 2024-03-04T16:27:39.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-21 2024-03-04T16:27:39.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:39.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:39.145 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:39.204 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:39.205 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:39.260+0000 7f90d072a800 1 bucket sync caught up with source: 2024-03-04T16:27:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.53.10, , 00000000003.31.10, , 00000000015.45.10, , , , , ] 2024-03-04T16:27:39.260 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.53.10, , 00000000003.31.10, , 00000000015.45.10, , , , , ] 2024-03-04T16:27:39.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:39.260+0000 7f90d072a800 0 bucket checkpoint complete 2024-03-04T16:27:39.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone2, test-zone1} 2024-03-04T16:27:39.279 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.279 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.279 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.279 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:39.279 DEBUG:boto:Data: 2024-03-04T16:27:39.280 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.280 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.280 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.280 DEBUG:boto:Params: {} 2024-03-04T16:27:39.280 DEBUG:boto:Token: None 2024-03-04T16:27:39.281 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pnFz7cNtuSXILuslD9WTBE+rGgQ= 2024-03-04T16:27:39.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pnFz7cNtuSXILuslD9WTBE+rGgQ=', 'Content-Length': '0'} 2024-03-04T16:27:39.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e95f37172edd11b-0065e5f67b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.285 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.285 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.285 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.285 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:39.285 DEBUG:boto:Data: 2024-03-04T16:27:39.285 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.285 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.285 DEBUG:boto:Params: {} 2024-03-04T16:27:39.286 DEBUG:boto:Token: None 2024-03-04T16:27:39.286 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pnFz7cNtuSXILuslD9WTBE+rGgQ= 2024-03-04T16:27:39.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pnFz7cNtuSXILuslD9WTBE+rGgQ=', 'Content-Length': '0'} 2024-03-04T16:27:39.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbadfcb2c89d7271-0065e5f67b-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:27:39.289 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.289 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.289 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.289 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.289 DEBUG:boto:Method: GET 2024-03-04T16:27:39.289 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.289 DEBUG:boto:Data: 2024-03-04T16:27:39.289 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.290 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.290 DEBUG:boto:Params: {} 2024-03-04T16:27:39.290 DEBUG:boto:Token: None 2024-03-04T16:27:39.290 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/?versions 2024-03-04T16:27:39.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco= 2024-03-04T16:27:39.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco=', 'Content-Length': '0'} 2024-03-04T16:27:39.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8a4299a40c19547-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.293 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:39.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:39.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:27:39.294 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.295 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.295 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.295 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.295 DEBUG:boto:Method: GET 2024-03-04T16:27:39.295 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.295 DEBUG:boto:Data: 2024-03-04T16:27:39.295 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.295 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.295 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.295 DEBUG:boto:Params: {} 2024-03-04T16:27:39.295 DEBUG:boto:Token: None 2024-03-04T16:27:39.296 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/?versions 2024-03-04T16:27:39.296 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco= 2024-03-04T16:27:39.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco=', 'Content-Length': '0'} 2024-03-04T16:27:39.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d68320491b8cd53-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.298 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.298 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:39.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:39.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:39.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:39.299 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.299 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.299 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.299 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.299 DEBUG:boto:Method: GET 2024-03-04T16:27:39.299 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.299 DEBUG:boto:Data: 2024-03-04T16:27:39.300 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.300 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.300 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.300 DEBUG:boto:Params: {} 2024-03-04T16:27:39.300 DEBUG:boto:Token: None 2024-03-04T16:27:39.300 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/?versions 2024-03-04T16:27:39.300 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco= 2024-03-04T16:27:39.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco=', 'Content-Length': '0'} 2024-03-04T16:27:39.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071aaeb0ad1b04f04-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.303 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.303 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.303 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.303 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.303 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.303 DEBUG:boto:Method: GET 2024-03-04T16:27:39.304 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:39.304 DEBUG:boto:Data: 2024-03-04T16:27:39.304 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.304 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.304 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.304 DEBUG:boto:Params: {} 2024-03-04T16:27:39.304 DEBUG:boto:Token: None 2024-03-04T16:27:39.304 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/?versions 2024-03-04T16:27:39.304 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco= 2024-03-04T16:27:39.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+LUbpx79IQcdHJEllko09oSGtco=', 'Content-Length': '0'} 2024-03-04T16:27:39.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037504d4d14c57b4a-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.307 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:39.307 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.308 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.308 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.308 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.308 DEBUG:boto:Method: GET 2024-03-04T16:27:39.308 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.308 DEBUG:boto:Data: 2024-03-04T16:27:39.308 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.308 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.308 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.308 DEBUG:boto:Params: {} 2024-03-04T16:27:39.308 DEBUG:boto:Token: None 2024-03-04T16:27:39.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.309 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af13b6af32311c5b-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.311 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.311 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.311 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.311 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.311 DEBUG:boto:Method: GET 2024-03-04T16:27:39.311 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.311 DEBUG:boto:Data: 2024-03-04T16:27:39.311 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.311 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.312 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.312 DEBUG:boto:Params: {} 2024-03-04T16:27:39.312 DEBUG:boto:Token: None 2024-03-04T16:27:39.312 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.312 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043d8a7dc7a87a76d-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.314 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.314 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.314 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.314 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.315 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.315 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.315 DEBUG:boto:Data: 2024-03-04T16:27:39.315 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.315 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.315 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.315 DEBUG:boto:Params: {} 2024-03-04T16:27:39.315 DEBUG:boto:Token: None 2024-03-04T16:27:39.315 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q+UGjPX9iZp3+aInY9CdXkhNuRI= 2024-03-04T16:27:39.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q+UGjPX9iZp3+aInY9CdXkhNuRI=', 'Content-Length': '0'} 2024-03-04T16:27:39.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000429137788d5dfd35-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.317 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.317 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.318 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.318 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.318 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.318 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.318 DEBUG:boto:Data: 2024-03-04T16:27:39.318 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.318 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.318 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.318 DEBUG:boto:Params: {} 2024-03-04T16:27:39.318 DEBUG:boto:Token: None 2024-03-04T16:27:39.319 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.319 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:q+UGjPX9iZp3+aInY9CdXkhNuRI= 2024-03-04T16:27:39.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:q+UGjPX9iZp3+aInY9CdXkhNuRI=', 'Content-Length': '0'} 2024-03-04T16:27:39.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa95fe9db0aa46d8-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:39.321 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.321 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.321 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.321 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.321 DEBUG:boto:Method: GET 2024-03-04T16:27:39.321 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.321 DEBUG:boto:Data: 2024-03-04T16:27:39.321 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.321 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.321 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.322 DEBUG:boto:Params: {} 2024-03-04T16:27:39.322 DEBUG:boto:Token: None 2024-03-04T16:27:39.322 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.322 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a94e0d08613dd8cf-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.324 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.324 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.324 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.324 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.324 DEBUG:boto:Method: GET 2024-03-04T16:27:39.324 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.324 DEBUG:boto:Data: 2024-03-04T16:27:39.324 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.325 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.325 DEBUG:boto:Params: {} 2024-03-04T16:27:39.325 DEBUG:boto:Token: None 2024-03-04T16:27:39.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.325 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000236b19108ac957e5-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.327 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.327 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.327 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.327 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.327 DEBUG:boto:Method: GET 2024-03-04T16:27:39.327 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.328 DEBUG:boto:Data: 2024-03-04T16:27:39.328 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.328 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.328 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.328 DEBUG:boto:Params: {} 2024-03-04T16:27:39.328 DEBUG:boto:Token: None 2024-03-04T16:27:39.328 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.328 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca828c07d767f286-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.372 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.373 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.373 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.373 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.373 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.373 DEBUG:boto:Method: GET 2024-03-04T16:27:39.373 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.373 DEBUG:boto:Data: 2024-03-04T16:27:39.373 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.373 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.374 DEBUG:boto:Params: {} 2024-03-04T16:27:39.374 DEBUG:boto:Token: None 2024-03-04T16:27:39.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.374 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbb4fbbfdf4fdaf-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.376 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.377 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.377 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.377 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.377 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:39.377 DEBUG:boto:Data: 2024-03-04T16:27:39.377 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.377 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.377 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.378 DEBUG:boto:Params: {} 2024-03-04T16:27:39.378 DEBUG:boto:Token: None 2024-03-04T16:27:39.378 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy 2024-03-04T16:27:39.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qLwK2ucnEaoHnGCpzFQHYC8SUJA= 2024-03-04T16:27:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qLwK2ucnEaoHnGCpzFQHYC8SUJA=', 'Content-Length': '0'} 2024-03-04T16:27:39.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096e86fee10798b21-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.380 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.380 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.380 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.380 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:39.380 DEBUG:boto:Data: 2024-03-04T16:27:39.381 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.381 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.381 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.381 DEBUG:boto:Params: {} 2024-03-04T16:27:39.381 DEBUG:boto:Token: None 2024-03-04T16:27:39.381 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy 2024-03-04T16:27:39.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qLwK2ucnEaoHnGCpzFQHYC8SUJA= 2024-03-04T16:27:39.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qLwK2ucnEaoHnGCpzFQHYC8SUJA=', 'Content-Length': '0'} 2024-03-04T16:27:39.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb4107c247c84785-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:39.383 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.384 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.384 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.384 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.384 DEBUG:boto:Method: GET 2024-03-04T16:27:39.384 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.384 DEBUG:boto:Data: 2024-03-04T16:27:39.384 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.384 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.384 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.384 DEBUG:boto:Params: {} 2024-03-04T16:27:39.384 DEBUG:boto:Token: None 2024-03-04T16:27:39.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bd95dddc7e02793-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.387 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:39.387 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:39.387 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.387 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.387 DEBUG:boto:Method: GET 2024-03-04T16:27:39.387 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.387 DEBUG:boto:Data: 2024-03-04T16:27:39.387 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.387 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.387 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.387 DEBUG:boto:Params: {} 2024-03-04T16:27:39.388 DEBUG:boto:Token: None 2024-03-04T16:27:39.388 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:39.388 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc= 2024-03-04T16:27:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RIZOl50KBU/ZeQIwSXAwrKVhgbc=', 'Content-Length': '0'} 2024-03-04T16:27:39.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084aee7235ad80cc1-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.390 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.390 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.390 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.390 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.390 DEBUG:boto:Method: GET 2024-03-04T16:27:39.390 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.390 DEBUG:boto:Data: 2024-03-04T16:27:39.390 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.390 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.391 DEBUG:boto:Params: {} 2024-03-04T16:27:39.391 DEBUG:boto:Token: None 2024-03-04T16:27:39.391 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30ce6905bba62b6-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.393 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.394 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.394 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.394 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.394 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.394 DEBUG:boto:Method: GET 2024-03-04T16:27:39.394 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:39.394 DEBUG:boto:Data: 2024-03-04T16:27:39.394 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.394 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.394 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.395 DEBUG:boto:Params: {} 2024-03-04T16:27:39.395 DEBUG:boto:Token: None 2024-03-04T16:27:39.395 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5b1d4d0658a5e2-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.440 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:39.441 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.441 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.441 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.441 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.441 DEBUG:boto:Method: GET 2024-03-04T16:27:39.441 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.441 DEBUG:boto:Data: 2024-03-04T16:27:39.441 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.442 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.442 DEBUG:boto:Params: {} 2024-03-04T16:27:39.442 DEBUG:boto:Token: None 2024-03-04T16:27:39.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cb42f1e003257c3-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.484 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.484 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.485 DEBUG:boto:Method: GET 2024-03-04T16:27:39.485 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.485 DEBUG:boto:Data: 2024-03-04T16:27:39.485 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.485 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.485 DEBUG:boto:Params: {} 2024-03-04T16:27:39.485 DEBUG:boto:Token: None 2024-03-04T16:27:39.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a988929d6483b51d-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.488 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.489 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.489 DEBUG:boto:Data: 2024-03-04T16:27:39.489 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.489 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.489 DEBUG:boto:Params: {} 2024-03-04T16:27:39.489 DEBUG:boto:Token: None 2024-03-04T16:27:39.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HL6E6jy0jvdBsxPuVHz4emInLYk= 2024-03-04T16:27:39.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HL6E6jy0jvdBsxPuVHz4emInLYk=', 'Content-Length': '0'} 2024-03-04T16:27:39.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5520c33a5f94caf-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.491 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.492 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.492 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.492 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.492 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.492 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.492 DEBUG:boto:Data: 2024-03-04T16:27:39.492 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.492 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.492 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.493 DEBUG:boto:Params: {} 2024-03-04T16:27:39.493 DEBUG:boto:Token: None 2024-03-04T16:27:39.493 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HL6E6jy0jvdBsxPuVHz4emInLYk= 2024-03-04T16:27:39.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HL6E6jy0jvdBsxPuVHz4emInLYk=', 'Content-Length': '0'} 2024-03-04T16:27:39.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000170d68740653e6d3-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:39.495 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.495 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.496 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.496 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.496 DEBUG:boto:Method: GET 2024-03-04T16:27:39.496 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.496 DEBUG:boto:Data: 2024-03-04T16:27:39.496 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.496 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.496 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.496 DEBUG:boto:Params: {} 2024-03-04T16:27:39.496 DEBUG:boto:Token: None 2024-03-04T16:27:39.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c43f3b3a95996832-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.540 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.540 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.540 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.540 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.541 DEBUG:boto:Method: GET 2024-03-04T16:27:39.541 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.541 DEBUG:boto:Data: 2024-03-04T16:27:39.541 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.541 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.541 DEBUG:boto:Params: {} 2024-03-04T16:27:39.541 DEBUG:boto:Token: None 2024-03-04T16:27:39.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e1671932e9bafa6-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.544 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.544 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.544 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.544 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.544 DEBUG:boto:Method: GET 2024-03-04T16:27:39.544 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.544 DEBUG:boto:Data: 2024-03-04T16:27:39.544 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.545 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.545 DEBUG:boto:Params: {} 2024-03-04T16:27:39.545 DEBUG:boto:Token: None 2024-03-04T16:27:39.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c88f550e51f81e41-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.588 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.588 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.589 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.589 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.589 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.589 DEBUG:boto:Method: GET 2024-03-04T16:27:39.589 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.589 DEBUG:boto:Data: 2024-03-04T16:27:39.589 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.589 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.589 DEBUG:boto:Params: {} 2024-03-04T16:27:39.589 DEBUG:boto:Token: None 2024-03-04T16:27:39.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006b44eacb4614be3-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.592 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.593 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.593 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.593 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.593 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.593 DEBUG:boto:Data: 2024-03-04T16:27:39.593 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.593 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.593 DEBUG:boto:Params: {} 2024-03-04T16:27:39.594 DEBUG:boto:Token: None 2024-03-04T16:27:39.594 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JJFWtyy1vzinLmNGhm7Albir2Hw= 2024-03-04T16:27:39.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JJFWtyy1vzinLmNGhm7Albir2Hw=', 'Content-Length': '0'} 2024-03-04T16:27:39.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000598c1f3d967ee2b9-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.597 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.597 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.597 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.597 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.597 DEBUG:boto:Data: 2024-03-04T16:27:39.597 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.597 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.597 DEBUG:boto:Params: {} 2024-03-04T16:27:39.597 DEBUG:boto:Token: None 2024-03-04T16:27:39.598 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JJFWtyy1vzinLmNGhm7Albir2Hw= 2024-03-04T16:27:39.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JJFWtyy1vzinLmNGhm7Albir2Hw=', 'Content-Length': '0'} 2024-03-04T16:27:39.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003040f0df2e08724-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:39.600 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.600 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.600 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.600 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.600 DEBUG:boto:Method: GET 2024-03-04T16:27:39.600 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.601 DEBUG:boto:Data: 2024-03-04T16:27:39.601 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.601 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.601 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.601 DEBUG:boto:Params: {} 2024-03-04T16:27:39.601 DEBUG:boto:Token: None 2024-03-04T16:27:39.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002afd72ee1f13b230-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.644 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.644 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:39.644 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.644 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.645 DEBUG:boto:Method: GET 2024-03-04T16:27:39.645 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.645 DEBUG:boto:Data: 2024-03-04T16:27:39.645 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.645 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.645 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.645 DEBUG:boto:Params: {} 2024-03-04T16:27:39.645 DEBUG:boto:Token: None 2024-03-04T16:27:39.645 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:39.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g= 2024-03-04T16:27:39.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aF5bAnmqMYrmvcxmhSyyCPgDb2g=', 'Content-Length': '0'} 2024-03-04T16:27:39.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ade9e8177763dcdf-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.648 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.648 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.648 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.648 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.648 DEBUG:boto:Method: GET 2024-03-04T16:27:39.648 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.649 DEBUG:boto:Data: 2024-03-04T16:27:39.649 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.649 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.649 DEBUG:boto:Params: {} 2024-03-04T16:27:39.649 DEBUG:boto:Token: None 2024-03-04T16:27:39.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1cf22b444a4d731-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.692 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.693 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.693 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.693 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.693 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.693 DEBUG:boto:Method: GET 2024-03-04T16:27:39.693 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:39.693 DEBUG:boto:Data: 2024-03-04T16:27:39.693 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.693 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.693 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.693 DEBUG:boto:Params: {} 2024-03-04T16:27:39.694 DEBUG:boto:Token: None 2024-03-04T16:27:39.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2dac190644c4e0a-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.696 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:39.697 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.697 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.697 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.697 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.697 DEBUG:boto:Method: GET 2024-03-04T16:27:39.697 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.698 DEBUG:boto:Data: 2024-03-04T16:27:39.698 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.698 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.698 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.698 DEBUG:boto:Params: {} 2024-03-04T16:27:39.698 DEBUG:boto:Token: None 2024-03-04T16:27:39.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0563ff01baa1408-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.701 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.701 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.701 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.701 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.701 DEBUG:boto:Method: GET 2024-03-04T16:27:39.701 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.701 DEBUG:boto:Data: 2024-03-04T16:27:39.701 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.701 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.701 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.702 DEBUG:boto:Params: {} 2024-03-04T16:27:39.702 DEBUG:boto:Token: None 2024-03-04T16:27:39.702 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.702 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2f688464302f12c-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.748 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.749 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.749 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.749 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.749 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.749 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.749 DEBUG:boto:Data: 2024-03-04T16:27:39.749 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.750 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.750 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.750 DEBUG:boto:Params: {} 2024-03-04T16:27:39.750 DEBUG:boto:Token: None 2024-03-04T16:27:39.750 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.750 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:71uY10uuus18He+kAUYdo3nB+uk= 2024-03-04T16:27:39.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:71uY10uuus18He+kAUYdo3nB+uk=', 'Content-Length': '0'} 2024-03-04T16:27:39.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041875c1a35433753-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.754 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.754 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.754 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.754 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.754 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.754 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.754 DEBUG:boto:Data: 2024-03-04T16:27:39.754 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.754 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.754 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.755 DEBUG:boto:Params: {} 2024-03-04T16:27:39.755 DEBUG:boto:Token: None 2024-03-04T16:27:39.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:71uY10uuus18He+kAUYdo3nB+uk= 2024-03-04T16:27:39.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:71uY10uuus18He+kAUYdo3nB+uk=', 'Content-Length': '0'} 2024-03-04T16:27:39.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb7a4b7a65fce21d-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:39.757 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.757 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.757 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.757 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.757 DEBUG:boto:Method: GET 2024-03-04T16:27:39.758 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.758 DEBUG:boto:Data: 2024-03-04T16:27:39.758 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.758 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.758 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.758 DEBUG:boto:Params: {} 2024-03-04T16:27:39.758 DEBUG:boto:Token: None 2024-03-04T16:27:39.758 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.759 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f39a9f408a5ad0b-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.761 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.761 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.761 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.761 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.761 DEBUG:boto:Method: GET 2024-03-04T16:27:39.762 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.762 DEBUG:boto:Data: 2024-03-04T16:27:39.762 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.762 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.762 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.762 DEBUG:boto:Params: {} 2024-03-04T16:27:39.762 DEBUG:boto:Token: None 2024-03-04T16:27:39.763 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.763 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000341f21a3ff375e2a-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.808 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.808 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.808 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.808 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.809 DEBUG:boto:Method: GET 2024-03-04T16:27:39.809 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.809 DEBUG:boto:Data: 2024-03-04T16:27:39.809 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.809 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.809 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.809 DEBUG:boto:Params: {} 2024-03-04T16:27:39.809 DEBUG:boto:Token: None 2024-03-04T16:27:39.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.810 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045906dce03f1757f-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.812 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.813 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.813 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.813 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.813 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.813 DEBUG:boto:Method: GET 2024-03-04T16:27:39.813 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.813 DEBUG:boto:Data: 2024-03-04T16:27:39.813 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.813 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.813 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.813 DEBUG:boto:Params: {} 2024-03-04T16:27:39.814 DEBUG:boto:Token: None 2024-03-04T16:27:39.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6a9253b5b10f1e-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.817 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.817 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.818 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.818 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.818 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.818 DEBUG:boto:Data: 2024-03-04T16:27:39.818 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.819 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.819 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.819 DEBUG:boto:Params: {} 2024-03-04T16:27:39.819 DEBUG:boto:Token: None 2024-03-04T16:27:39.819 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.819 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:14bX6EVj6+j3+jNaDWbDa7o82uI= 2024-03-04T16:27:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:14bX6EVj6+j3+jNaDWbDa7o82uI=', 'Content-Length': '0'} 2024-03-04T16:27:39.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008358f2d27bfa3a9e-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.821 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.821 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.822 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.822 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.822 DEBUG:boto:Data: 2024-03-04T16:27:39.822 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.822 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.822 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.822 DEBUG:boto:Params: {} 2024-03-04T16:27:39.822 DEBUG:boto:Token: None 2024-03-04T16:27:39.823 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.823 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:14bX6EVj6+j3+jNaDWbDa7o82uI= 2024-03-04T16:27:39.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:14bX6EVj6+j3+jNaDWbDa7o82uI=', 'Content-Length': '0'} 2024-03-04T16:27:39.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002109ff912aabb819-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:39.825 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.825 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.826 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.826 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.826 DEBUG:boto:Method: GET 2024-03-04T16:27:39.826 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.826 DEBUG:boto:Data: 2024-03-04T16:27:39.826 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.826 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.826 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.826 DEBUG:boto:Params: {} 2024-03-04T16:27:39.826 DEBUG:boto:Token: None 2024-03-04T16:27:39.827 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.827 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e51aaeda6133d626-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.872 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.872 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:39.872 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.873 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.873 DEBUG:boto:Method: GET 2024-03-04T16:27:39.873 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.873 DEBUG:boto:Data: 2024-03-04T16:27:39.873 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.873 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.873 DEBUG:boto:Params: {} 2024-03-04T16:27:39.873 DEBUG:boto:Token: None 2024-03-04T16:27:39.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:39.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao= 2024-03-04T16:27:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nWRwgl+csNdc6MBpDM+IVV3ymao=', 'Content-Length': '0'} 2024-03-04T16:27:39.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb95ce41e1dd3946-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.916 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.916 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.916 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.916 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.917 DEBUG:boto:Method: GET 2024-03-04T16:27:39.917 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.917 DEBUG:boto:Data: 2024-03-04T16:27:39.917 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.917 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.917 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.917 DEBUG:boto:Params: {} 2024-03-04T16:27:39.917 DEBUG:boto:Token: None 2024-03-04T16:27:39.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.917 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3fe97ffcf7c9c4-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.920 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.920 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:39.921 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:39.921 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.921 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.921 DEBUG:boto:Method: GET 2024-03-04T16:27:39.921 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:39.921 DEBUG:boto:Data: 2024-03-04T16:27:39.921 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.921 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.921 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.921 DEBUG:boto:Params: {} 2024-03-04T16:27:39.921 DEBUG:boto:Token: None 2024-03-04T16:27:39.922 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/ 2024-03-04T16:27:39.922 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c= 2024-03-04T16:27:39.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wzXU/9ZotM+P8njELqWOfOZGp1c=', 'Content-Length': '0'} 2024-03-04T16:27:39.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c117391332ee8f0a-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.924 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:39.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:39.924 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.925 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.925 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.925 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.925 DEBUG:boto:Method: GET 2024-03-04T16:27:39.925 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.925 DEBUG:boto:Data: 2024-03-04T16:27:39.925 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.925 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.925 DEBUG:boto:Params: {} 2024-03-04T16:27:39.925 DEBUG:boto:Token: None 2024-03-04T16:27:39.926 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg= 2024-03-04T16:27:39.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg=', 'Content-Length': '0'} 2024-03-04T16:27:39.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de4ddd3c1abb5d81-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.968 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.968 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.969 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.969 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.969 DEBUG:boto:Method: GET 2024-03-04T16:27:39.969 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.969 DEBUG:boto:Data: 2024-03-04T16:27:39.969 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.969 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.969 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.969 DEBUG:boto:Params: {} 2024-03-04T16:27:39.970 DEBUG:boto:Token: None 2024-03-04T16:27:39.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg= 2024-03-04T16:27:39.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg=', 'Content-Length': '0'} 2024-03-04T16:27:39.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baeb7e1a899cd014-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.972 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.972 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.972 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.973 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.973 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.973 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.973 DEBUG:boto:Data: 2024-03-04T16:27:39.973 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.973 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.973 DEBUG:boto:Params: {} 2024-03-04T16:27:39.974 DEBUG:boto:Token: None 2024-03-04T16:27:39.974 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VIPMss25hHPWMDee8Lox91a9QZE= 2024-03-04T16:27:39.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VIPMss25hHPWMDee8Lox91a9QZE=', 'Content-Length': '0'} 2024-03-04T16:27:39.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000429a79814cd56d33-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.976 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.977 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.977 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.977 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.977 DEBUG:boto:Method: HEAD 2024-03-04T16:27:39.977 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.977 DEBUG:boto:Data: 2024-03-04T16:27:39.977 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:39.978 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.978 DEBUG:boto:Params: {} 2024-03-04T16:27:39.978 DEBUG:boto:Token: None 2024-03-04T16:27:39.978 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VIPMss25hHPWMDee8Lox91a9QZE= 2024-03-04T16:27:39.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VIPMss25hHPWMDee8Lox91a9QZE=', 'Content-Length': '0'} 2024-03-04T16:27:39.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ec7700cb3fdce73-0065e5f67b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:39.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:39.981 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.981 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:39.981 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.981 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.981 DEBUG:boto:Method: GET 2024-03-04T16:27:39.981 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.981 DEBUG:boto:Data: 2024-03-04T16:27:39.982 DEBUG:boto:Headers: {} 2024-03-04T16:27:39.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:39.982 DEBUG:boto:Port: 8000 2024-03-04T16:27:39.982 DEBUG:boto:Params: {} 2024-03-04T16:27:39.982 DEBUG:boto:Token: None 2024-03-04T16:27:39.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:39 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:39.982 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg= 2024-03-04T16:27:39.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6kepmCjjX8hqqkvC4iflM1gRdzg=', 'Content-Length': '0'} 2024-03-04T16:27:39.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1400bc5828d412f-0065e5f67b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.028 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.028 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.029 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.029 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.029 DEBUG:boto:Method: GET 2024-03-04T16:27:40.029 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.029 DEBUG:boto:Data: 2024-03-04T16:27:40.030 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.030 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.030 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.030 DEBUG:boto:Params: {} 2024-03-04T16:27:40.031 DEBUG:boto:Token: None 2024-03-04T16:27:40.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.031 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY= 2024-03-04T16:27:40.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY=', 'Content-Length': '0'} 2024-03-04T16:27:40.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6d86b4399765b58-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.076 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.077 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.077 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.077 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.077 DEBUG:boto:Method: GET 2024-03-04T16:27:40.077 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.078 DEBUG:boto:Data: 2024-03-04T16:27:40.078 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.078 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:40.078 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.078 DEBUG:boto:Params: {} 2024-03-04T16:27:40.079 DEBUG:boto:Token: None 2024-03-04T16:27:40.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.079 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060a0b831acb83e2a-0065e5f67c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.082 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.083 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.083 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.083 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.084 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.084 DEBUG:boto:Method: GET 2024-03-04T16:27:40.084 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.084 DEBUG:boto:Data: 2024-03-04T16:27:40.084 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.085 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.085 DEBUG:boto:Params: {} 2024-03-04T16:27:40.085 DEBUG:boto:Token: None 2024-03-04T16:27:40.086 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.086 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb16d86ea599568-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.132 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.133 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.133 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.133 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.133 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.133 DEBUG:boto:Data: 2024-03-04T16:27:40.133 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.133 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:40.133 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.133 DEBUG:boto:Params: {} 2024-03-04T16:27:40.133 DEBUG:boto:Token: None 2024-03-04T16:27:40.134 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.134 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gzIo4RW5spZb0oc4VrQ5+V/Oxy4= 2024-03-04T16:27:40.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gzIo4RW5spZb0oc4VrQ5+V/Oxy4=', 'Content-Length': '0'} 2024-03-04T16:27:40.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b5005c33815c736-0065e5f67c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.136 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.136 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.137 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.137 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.137 DEBUG:boto:Data: 2024-03-04T16:27:40.137 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.137 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.137 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.137 DEBUG:boto:Params: {} 2024-03-04T16:27:40.137 DEBUG:boto:Token: None 2024-03-04T16:27:40.137 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.138 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gzIo4RW5spZb0oc4VrQ5+V/Oxy4= 2024-03-04T16:27:40.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gzIo4RW5spZb0oc4VrQ5+V/Oxy4=', 'Content-Length': '0'} 2024-03-04T16:27:40.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d763e2a55fdda1c3-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:40.141 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.141 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.141 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.141 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.141 DEBUG:boto:Method: GET 2024-03-04T16:27:40.141 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.142 DEBUG:boto:Data: 2024-03-04T16:27:40.142 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.142 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:40.142 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.142 DEBUG:boto:Params: {} 2024-03-04T16:27:40.142 DEBUG:boto:Token: None 2024-03-04T16:27:40.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.143 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY= 2024-03-04T16:27:40.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY=', 'Content-Length': '0'} 2024-03-04T16:27:40.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4e123017e2e0d6-0065e5f67c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.188 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.188 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:40.188 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.188 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.189 DEBUG:boto:Method: GET 2024-03-04T16:27:40.189 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.189 DEBUG:boto:Data: 2024-03-04T16:27:40.189 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.189 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.189 DEBUG:boto:Params: {} 2024-03-04T16:27:40.189 DEBUG:boto:Token: None 2024-03-04T16:27:40.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:40.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY= 2024-03-04T16:27:40.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2lC9kZbfHnBc1E8dEVGUAnYg8LY=', 'Content-Length': '0'} 2024-03-04T16:27:40.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdafbe5163341b87-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.236 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.236 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.236 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.236 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.236 DEBUG:boto:Method: GET 2024-03-04T16:27:40.237 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.237 DEBUG:boto:Data: 2024-03-04T16:27:40.237 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:40.237 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.237 DEBUG:boto:Params: {} 2024-03-04T16:27:40.237 DEBUG:boto:Token: None 2024-03-04T16:27:40.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208b6c6ac0c87af6-0065e5f67c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.240 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.241 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.241 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.241 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.241 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.241 DEBUG:boto:Method: GET 2024-03-04T16:27:40.241 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:40.241 DEBUG:boto:Data: 2024-03-04T16:27:40.241 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.241 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.241 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.241 DEBUG:boto:Params: {} 2024-03-04T16:27:40.242 DEBUG:boto:Token: None 2024-03-04T16:27:40.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb4a77c71edd2b1-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.244 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.244 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone2, test-zone1} 2024-03-04T16:27:40.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone3, test-zone1} 2024-03-04T16:27:40.245 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.245 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.245 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.245 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:40.245 DEBUG:boto:Data: 2024-03-04T16:27:40.245 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.245 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.245 DEBUG:boto:Params: {} 2024-03-04T16:27:40.245 DEBUG:boto:Token: None 2024-03-04T16:27:40.246 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5bU3mAaDcdQ830CFUr87W+ozQr8= 2024-03-04T16:27:40.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5bU3mAaDcdQ830CFUr87W+ozQr8=', 'Content-Length': '0'} 2024-03-04T16:27:40.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037cc863949cef5a7-0065e5f67c-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.248 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.248 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.248 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.248 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:40.248 DEBUG:boto:Data: 2024-03-04T16:27:40.249 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.249 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.249 DEBUG:boto:Params: {} 2024-03-04T16:27:40.249 DEBUG:boto:Token: None 2024-03-04T16:27:40.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5bU3mAaDcdQ830CFUr87W+ozQr8= 2024-03-04T16:27:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5bU3mAaDcdQ830CFUr87W+ozQr8=', 'Content-Length': '0'} 2024-03-04T16:27:40.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a80f0f36c92602d3-0065e5f67c-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.251 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:27:40.252 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.252 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.252 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.252 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.252 DEBUG:boto:Method: GET 2024-03-04T16:27:40.252 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.252 DEBUG:boto:Data: 2024-03-04T16:27:40.252 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.252 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.252 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.252 DEBUG:boto:Params: {} 2024-03-04T16:27:40.253 DEBUG:boto:Token: None 2024-03-04T16:27:40.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/?versions 2024-03-04T16:27:40.253 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI= 2024-03-04T16:27:40.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI=', 'Content-Length': '0'} 2024-03-04T16:27:40.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab48cf874540cee7-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.256 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.256 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:40.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:40.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:40.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:40.257 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:27:40.257 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.257 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.257 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.257 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.257 DEBUG:boto:Method: GET 2024-03-04T16:27:40.257 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.257 DEBUG:boto:Data: 2024-03-04T16:27:40.257 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.257 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.258 DEBUG:boto:Params: {} 2024-03-04T16:27:40.258 DEBUG:boto:Token: None 2024-03-04T16:27:40.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/?versions 2024-03-04T16:27:40.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI= 2024-03-04T16:27:40.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI=', 'Content-Length': '0'} 2024-03-04T16:27:40.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a87c9faaf9bd6271-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.261 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.261 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:40.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:40.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:40.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:40.262 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.262 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.262 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.262 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.262 DEBUG:boto:Method: GET 2024-03-04T16:27:40.262 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.262 DEBUG:boto:Data: 2024-03-04T16:27:40.262 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.262 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.262 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.262 DEBUG:boto:Params: {} 2024-03-04T16:27:40.263 DEBUG:boto:Token: None 2024-03-04T16:27:40.263 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/?versions 2024-03-04T16:27:40.263 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI= 2024-03-04T16:27:40.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI=', 'Content-Length': '0'} 2024-03-04T16:27:40.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4768e5635595b2-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.265 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.265 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.266 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.266 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.266 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.266 DEBUG:boto:Method: GET 2024-03-04T16:27:40.266 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:40.266 DEBUG:boto:Data: 2024-03-04T16:27:40.266 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.266 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.266 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.266 DEBUG:boto:Params: {} 2024-03-04T16:27:40.266 DEBUG:boto:Token: None 2024-03-04T16:27:40.267 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/?versions 2024-03-04T16:27:40.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI= 2024-03-04T16:27:40.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SHUYK37wA/FmkoFCCP4RySQ8yCI=', 'Content-Length': '0'} 2024-03-04T16:27:40.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000853fa682969d3d71-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.269 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:40.270 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.270 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.270 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.270 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.270 DEBUG:boto:Method: GET 2024-03-04T16:27:40.270 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.270 DEBUG:boto:Data: 2024-03-04T16:27:40.270 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.270 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.270 DEBUG:boto:Params: {} 2024-03-04T16:27:40.271 DEBUG:boto:Token: None 2024-03-04T16:27:40.271 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae589ebcbedb1802-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.273 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.273 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.273 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.273 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.273 DEBUG:boto:Method: GET 2024-03-04T16:27:40.273 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.273 DEBUG:boto:Data: 2024-03-04T16:27:40.274 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.274 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.274 DEBUG:boto:Params: {} 2024-03-04T16:27:40.274 DEBUG:boto:Token: None 2024-03-04T16:27:40.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000102d53f55f1137b5-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.276 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.276 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.276 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.277 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.277 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.277 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.277 DEBUG:boto:Data: 2024-03-04T16:27:40.277 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.277 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.277 DEBUG:boto:Params: {} 2024-03-04T16:27:40.277 DEBUG:boto:Token: None 2024-03-04T16:27:40.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.278 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iV1bDUQeEQaDPz56cdw67F0nrxo= 2024-03-04T16:27:40.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iV1bDUQeEQaDPz56cdw67F0nrxo=', 'Content-Length': '0'} 2024-03-04T16:27:40.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022fe8c56df657bb1-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.280 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.280 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.280 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.280 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.280 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.280 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.280 DEBUG:boto:Data: 2024-03-04T16:27:40.281 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.281 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.281 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.281 DEBUG:boto:Params: {} 2024-03-04T16:27:40.281 DEBUG:boto:Token: None 2024-03-04T16:27:40.281 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iV1bDUQeEQaDPz56cdw67F0nrxo= 2024-03-04T16:27:40.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iV1bDUQeEQaDPz56cdw67F0nrxo=', 'Content-Length': '0'} 2024-03-04T16:27:40.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bc94c20ea273f77-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:40.284 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.284 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.284 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.284 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.284 DEBUG:boto:Method: GET 2024-03-04T16:27:40.284 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.284 DEBUG:boto:Data: 2024-03-04T16:27:40.284 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.284 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.284 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.284 DEBUG:boto:Params: {} 2024-03-04T16:27:40.285 DEBUG:boto:Token: None 2024-03-04T16:27:40.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.285 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ade5b13e4e1af721-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.287 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.287 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.287 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.287 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.287 DEBUG:boto:Method: GET 2024-03-04T16:27:40.287 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.287 DEBUG:boto:Data: 2024-03-04T16:27:40.287 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.287 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.287 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.287 DEBUG:boto:Params: {} 2024-03-04T16:27:40.288 DEBUG:boto:Token: None 2024-03-04T16:27:40.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca8219939cb53f58-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.290 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.290 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.290 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.290 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.290 DEBUG:boto:Method: GET 2024-03-04T16:27:40.290 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.290 DEBUG:boto:Data: 2024-03-04T16:27:40.290 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.291 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.291 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.291 DEBUG:boto:Params: {} 2024-03-04T16:27:40.291 DEBUG:boto:Token: None 2024-03-04T16:27:40.291 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.291 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4fabd1a7f42f62b-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.340 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.341 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.341 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.341 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.341 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.341 DEBUG:boto:Method: GET 2024-03-04T16:27:40.341 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.342 DEBUG:boto:Data: 2024-03-04T16:27:40.342 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.342 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.342 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.342 DEBUG:boto:Params: {} 2024-03-04T16:27:40.342 DEBUG:boto:Token: None 2024-03-04T16:27:40.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2dc49b3cd7266c-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.345 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.345 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.345 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.346 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.346 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:40.346 DEBUG:boto:Data: 2024-03-04T16:27:40.346 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.346 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.346 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.346 DEBUG:boto:Params: {} 2024-03-04T16:27:40.346 DEBUG:boto:Token: None 2024-03-04T16:27:40.347 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy 2024-03-04T16:27:40.347 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h7+iBqHTXko+aaNAFj/VICz2W9w= 2024-03-04T16:27:40.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h7+iBqHTXko+aaNAFj/VICz2W9w=', 'Content-Length': '0'} 2024-03-04T16:27:40.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005953aaa722846d24-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.349 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.349 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.349 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.349 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:40.349 DEBUG:boto:Data: 2024-03-04T16:27:40.350 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.350 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.350 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.350 DEBUG:boto:Params: {} 2024-03-04T16:27:40.350 DEBUG:boto:Token: None 2024-03-04T16:27:40.350 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy 2024-03-04T16:27:40.351 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:h7+iBqHTXko+aaNAFj/VICz2W9w= 2024-03-04T16:27:40.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:h7+iBqHTXko+aaNAFj/VICz2W9w=', 'Content-Length': '0'} 2024-03-04T16:27:40.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025a3a46a9e70b030-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:40.353 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.353 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.353 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.353 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.354 DEBUG:boto:Method: GET 2024-03-04T16:27:40.354 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.354 DEBUG:boto:Data: 2024-03-04T16:27:40.354 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.354 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.354 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.354 DEBUG:boto:Params: {} 2024-03-04T16:27:40.354 DEBUG:boto:Token: None 2024-03-04T16:27:40.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd5532c2d39e4b5b-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.356 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:40.356 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:40.356 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.356 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.357 DEBUG:boto:Method: GET 2024-03-04T16:27:40.357 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.357 DEBUG:boto:Data: 2024-03-04T16:27:40.357 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.357 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.357 DEBUG:boto:Params: {} 2024-03-04T16:27:40.357 DEBUG:boto:Token: None 2024-03-04T16:27:40.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:40.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo= 2024-03-04T16:27:40.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8IIrLcXKmzlitXHm7/TR/7bRjeo=', 'Content-Length': '0'} 2024-03-04T16:27:40.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002379ce1015a8684f-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.360 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.360 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.360 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.361 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.361 DEBUG:boto:Method: GET 2024-03-04T16:27:40.361 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.361 DEBUG:boto:Data: 2024-03-04T16:27:40.361 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.361 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.361 DEBUG:boto:Params: {} 2024-03-04T16:27:40.362 DEBUG:boto:Token: None 2024-03-04T16:27:40.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c3494820f61431-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.367 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.367 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.367 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.368 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.368 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.368 DEBUG:boto:Method: GET 2024-03-04T16:27:40.368 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:40.368 DEBUG:boto:Data: 2024-03-04T16:27:40.368 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.369 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.369 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.369 DEBUG:boto:Params: {} 2024-03-04T16:27:40.369 DEBUG:boto:Token: None 2024-03-04T16:27:40.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6e6dc50e17b9927-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.373 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:40.374 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.374 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.374 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.374 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.375 DEBUG:boto:Method: GET 2024-03-04T16:27:40.375 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.375 DEBUG:boto:Data: 2024-03-04T16:27:40.375 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.375 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.375 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.375 DEBUG:boto:Params: {} 2024-03-04T16:27:40.375 DEBUG:boto:Token: None 2024-03-04T16:27:40.375 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.376 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d780517cf4f1d97-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.378 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.378 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.378 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.378 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.378 DEBUG:boto:Method: GET 2024-03-04T16:27:40.378 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.378 DEBUG:boto:Data: 2024-03-04T16:27:40.378 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.378 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.378 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.378 DEBUG:boto:Params: {} 2024-03-04T16:27:40.379 DEBUG:boto:Token: None 2024-03-04T16:27:40.379 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.379 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044709034e255d978-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.424 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.424 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.424 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.424 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.425 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.425 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.425 DEBUG:boto:Data: 2024-03-04T16:27:40.425 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.425 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.425 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.425 DEBUG:boto:Params: {} 2024-03-04T16:27:40.425 DEBUG:boto:Token: None 2024-03-04T16:27:40.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:anbqMdrA5rVn3aPorL53DAV4SKk= 2024-03-04T16:27:40.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:anbqMdrA5rVn3aPorL53DAV4SKk=', 'Content-Length': '0'} 2024-03-04T16:27:40.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060ef83e915be0806-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.428 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.428 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.428 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.428 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.428 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.428 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.429 DEBUG:boto:Data: 2024-03-04T16:27:40.429 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.429 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.429 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.429 DEBUG:boto:Params: {} 2024-03-04T16:27:40.429 DEBUG:boto:Token: None 2024-03-04T16:27:40.429 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:anbqMdrA5rVn3aPorL53DAV4SKk= 2024-03-04T16:27:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:anbqMdrA5rVn3aPorL53DAV4SKk=', 'Content-Length': '0'} 2024-03-04T16:27:40.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cffae2e600a4ae4e-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:40.431 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.431 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.431 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.432 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.432 DEBUG:boto:Method: GET 2024-03-04T16:27:40.432 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.432 DEBUG:boto:Data: 2024-03-04T16:27:40.432 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.432 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.432 DEBUG:boto:Params: {} 2024-03-04T16:27:40.432 DEBUG:boto:Token: None 2024-03-04T16:27:40.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003961b36d950fa44e-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.476 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.476 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.476 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.476 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.477 DEBUG:boto:Method: GET 2024-03-04T16:27:40.477 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.477 DEBUG:boto:Data: 2024-03-04T16:27:40.477 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.477 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.477 DEBUG:boto:Params: {} 2024-03-04T16:27:40.477 DEBUG:boto:Token: None 2024-03-04T16:27:40.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007238c9021347ddfb-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.480 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.480 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.480 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.480 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.480 DEBUG:boto:Method: GET 2024-03-04T16:27:40.480 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.480 DEBUG:boto:Data: 2024-03-04T16:27:40.480 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.480 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.480 DEBUG:boto:Params: {} 2024-03-04T16:27:40.481 DEBUG:boto:Token: None 2024-03-04T16:27:40.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c747a3a546bb1e-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.524 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.525 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.525 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.525 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.525 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.525 DEBUG:boto:Method: GET 2024-03-04T16:27:40.525 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.525 DEBUG:boto:Data: 2024-03-04T16:27:40.525 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.525 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.525 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.525 DEBUG:boto:Params: {} 2024-03-04T16:27:40.526 DEBUG:boto:Token: None 2024-03-04T16:27:40.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017d089fe98e5fc7-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.529 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.529 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.529 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.529 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.529 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.529 DEBUG:boto:Data: 2024-03-04T16:27:40.529 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.530 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.530 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.530 DEBUG:boto:Params: {} 2024-03-04T16:27:40.530 DEBUG:boto:Token: None 2024-03-04T16:27:40.530 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cpbNkSCnTeHJAkZXhjJ8eCdkzO8= 2024-03-04T16:27:40.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cpbNkSCnTeHJAkZXhjJ8eCdkzO8=', 'Content-Length': '0'} 2024-03-04T16:27:40.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef3e5b2c90b1e7e6-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.533 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.533 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.533 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.533 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.533 DEBUG:boto:Data: 2024-03-04T16:27:40.533 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.533 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.534 DEBUG:boto:Params: {} 2024-03-04T16:27:40.534 DEBUG:boto:Token: None 2024-03-04T16:27:40.534 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cpbNkSCnTeHJAkZXhjJ8eCdkzO8= 2024-03-04T16:27:40.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cpbNkSCnTeHJAkZXhjJ8eCdkzO8=', 'Content-Length': '0'} 2024-03-04T16:27:40.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b2793fbfa2c0e87-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:40.537 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.537 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.537 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.537 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.537 DEBUG:boto:Method: GET 2024-03-04T16:27:40.537 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.537 DEBUG:boto:Data: 2024-03-04T16:27:40.537 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.538 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.538 DEBUG:boto:Params: {} 2024-03-04T16:27:40.538 DEBUG:boto:Token: None 2024-03-04T16:27:40.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ee0ccaaf5b81258-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.588 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.588 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:40.588 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.588 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.589 DEBUG:boto:Method: GET 2024-03-04T16:27:40.589 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.589 DEBUG:boto:Data: 2024-03-04T16:27:40.589 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.589 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.589 DEBUG:boto:Params: {} 2024-03-04T16:27:40.589 DEBUG:boto:Token: None 2024-03-04T16:27:40.589 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:40.589 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU= 2024-03-04T16:27:40.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nslswv3UOfF+l7WkMDCOxowcqHU=', 'Content-Length': '0'} 2024-03-04T16:27:40.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f7182be0cb85d01-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.592 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.593 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.593 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.593 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.593 DEBUG:boto:Method: GET 2024-03-04T16:27:40.593 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.593 DEBUG:boto:Data: 2024-03-04T16:27:40.593 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.594 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.594 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.594 DEBUG:boto:Params: {} 2024-03-04T16:27:40.594 DEBUG:boto:Token: None 2024-03-04T16:27:40.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059988340714154a4-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.640 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.641 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.641 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.641 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.641 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.641 DEBUG:boto:Method: GET 2024-03-04T16:27:40.641 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:40.641 DEBUG:boto:Data: 2024-03-04T16:27:40.641 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.641 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.641 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.642 DEBUG:boto:Params: {} 2024-03-04T16:27:40.642 DEBUG:boto:Token: None 2024-03-04T16:27:40.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007137de7450fda78d-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.645 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:40.646 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.646 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.646 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.646 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.646 DEBUG:boto:Method: GET 2024-03-04T16:27:40.646 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.646 DEBUG:boto:Data: 2024-03-04T16:27:40.646 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.647 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.647 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.647 DEBUG:boto:Params: {} 2024-03-04T16:27:40.647 DEBUG:boto:Token: None 2024-03-04T16:27:40.647 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.647 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd4a7571d1a2184e-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.692 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.692 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.692 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.693 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.693 DEBUG:boto:Method: GET 2024-03-04T16:27:40.693 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.693 DEBUG:boto:Data: 2024-03-04T16:27:40.693 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.693 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.693 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.693 DEBUG:boto:Params: {} 2024-03-04T16:27:40.693 DEBUG:boto:Token: None 2024-03-04T16:27:40.693 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c4e1494c342e3f-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.740 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.740 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.741 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.741 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.741 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.741 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.741 DEBUG:boto:Data: 2024-03-04T16:27:40.742 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.742 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.742 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.742 DEBUG:boto:Params: {} 2024-03-04T16:27:40.742 DEBUG:boto:Token: None 2024-03-04T16:27:40.743 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.743 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OQhqGLtGOVPNLRcHrYWUMeirM6E= 2024-03-04T16:27:40.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OQhqGLtGOVPNLRcHrYWUMeirM6E=', 'Content-Length': '0'} 2024-03-04T16:27:40.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005030603e6f3d266a-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.746 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.746 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.746 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.746 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.747 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.747 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.747 DEBUG:boto:Data: 2024-03-04T16:27:40.747 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.747 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.748 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.748 DEBUG:boto:Params: {} 2024-03-04T16:27:40.748 DEBUG:boto:Token: None 2024-03-04T16:27:40.748 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.749 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OQhqGLtGOVPNLRcHrYWUMeirM6E= 2024-03-04T16:27:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OQhqGLtGOVPNLRcHrYWUMeirM6E=', 'Content-Length': '0'} 2024-03-04T16:27:40.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001108cd266d8653c0-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:40.752 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.752 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.752 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.752 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.753 DEBUG:boto:Method: GET 2024-03-04T16:27:40.753 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.753 DEBUG:boto:Data: 2024-03-04T16:27:40.753 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.753 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.753 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.754 DEBUG:boto:Params: {} 2024-03-04T16:27:40.754 DEBUG:boto:Token: None 2024-03-04T16:27:40.754 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.754 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f473ef347ceecf8e-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.804 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.804 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.804 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.804 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.805 DEBUG:boto:Method: GET 2024-03-04T16:27:40.805 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.805 DEBUG:boto:Data: 2024-03-04T16:27:40.805 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.805 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.805 DEBUG:boto:Params: {} 2024-03-04T16:27:40.805 DEBUG:boto:Token: None 2024-03-04T16:27:40.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063990f289d01c99c-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.848 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.848 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.848 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.848 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.849 DEBUG:boto:Method: GET 2024-03-04T16:27:40.849 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.849 DEBUG:boto:Data: 2024-03-04T16:27:40.849 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.849 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.849 DEBUG:boto:Params: {} 2024-03-04T16:27:40.849 DEBUG:boto:Token: None 2024-03-04T16:27:40.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f52e4b118fcf752f-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.900 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.901 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:40.901 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:40.901 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.901 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.901 DEBUG:boto:Method: GET 2024-03-04T16:27:40.901 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:40.901 DEBUG:boto:Data: 2024-03-04T16:27:40.901 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.901 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.901 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.901 DEBUG:boto:Params: {} 2024-03-04T16:27:40.902 DEBUG:boto:Token: None 2024-03-04T16:27:40.902 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/ 2024-03-04T16:27:40.902 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM= 2024-03-04T16:27:40.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7/ZW2xMfi3YiyaCGYHZvGGEJhXM=', 'Content-Length': '0'} 2024-03-04T16:27:40.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c65a2d8b7b67c81-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.905 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:40.906 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.906 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.906 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.906 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.906 DEBUG:boto:Data: 2024-03-04T16:27:40.907 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.907 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.907 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.907 DEBUG:boto:Params: {} 2024-03-04T16:27:40.907 DEBUG:boto:Token: None 2024-03-04T16:27:40.908 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.908 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/v3FV/R4SWHF0txqx/5drBTTxa4= 2024-03-04T16:27:40.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/v3FV/R4SWHF0txqx/5drBTTxa4=', 'Content-Length': '0'} 2024-03-04T16:27:40.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7a2047f5d260970-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.911 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.911 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.912 DEBUG:boto:Method: HEAD 2024-03-04T16:27:40.912 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.912 DEBUG:boto:Data: 2024-03-04T16:27:40.912 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.912 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.913 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.913 DEBUG:boto:Params: {} 2024-03-04T16:27:40.913 DEBUG:boto:Token: None 2024-03-04T16:27:40.913 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/v3FV/R4SWHF0txqx/5drBTTxa4= 2024-03-04T16:27:40.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/v3FV/R4SWHF0txqx/5drBTTxa4=', 'Content-Length': '0'} 2024-03-04T16:27:40.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eef400d5eef98719-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:40.916 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.917 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.917 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.917 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.917 DEBUG:boto:Method: GET 2024-03-04T16:27:40.917 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.917 DEBUG:boto:Data: 2024-03-04T16:27:40.917 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.917 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:40.917 DEBUG:boto:Port: 8001 2024-03-04T16:27:40.917 DEBUG:boto:Params: {} 2024-03-04T16:27:40.918 DEBUG:boto:Token: None 2024-03-04T16:27:40.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.918 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2f4b7b334c15d43-0065e5f67c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:40.960 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.960 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:40.961 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.961 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.961 DEBUG:boto:Method: GET 2024-03-04T16:27:40.961 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.961 DEBUG:boto:Data: 2024-03-04T16:27:40.961 DEBUG:boto:Headers: {} 2024-03-04T16:27:40.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:40.961 DEBUG:boto:Port: 8000 2024-03-04T16:27:40.962 DEBUG:boto:Params: {} 2024-03-04T16:27:40.962 DEBUG:boto:Token: None 2024-03-04T16:27:40.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:40 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:40.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo= 2024-03-04T16:27:40.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TcPxYMHEqTUhQCbmdJS4S9iCENo=', 'Content-Length': '0'} 2024-03-04T16:27:40.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001276d588a59ee78b-0065e5f67c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.008 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.008 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.008 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.009 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.009 DEBUG:boto:Method: GET 2024-03-04T16:27:41.009 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.009 DEBUG:boto:Data: 2024-03-04T16:27:41.009 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.009 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.010 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.010 DEBUG:boto:Params: {} 2024-03-04T16:27:41.010 DEBUG:boto:Token: None 2024-03-04T16:27:41.010 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d4867e30430b1d-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.014 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.014 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.014 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.014 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.015 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.015 DEBUG:boto:Method: GET 2024-03-04T16:27:41.015 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.015 DEBUG:boto:Data: 2024-03-04T16:27:41.015 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.015 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.015 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.015 DEBUG:boto:Params: {} 2024-03-04T16:27:41.015 DEBUG:boto:Token: None 2024-03-04T16:27:41.016 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.016 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8171ceeeab3725-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.018 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:41.019 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.019 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.019 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.019 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.019 DEBUG:boto:Method: GET 2024-03-04T16:27:41.019 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.020 DEBUG:boto:Data: 2024-03-04T16:27:41.020 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.020 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.020 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.020 DEBUG:boto:Params: {} 2024-03-04T16:27:41.020 DEBUG:boto:Token: None 2024-03-04T16:27:41.020 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dc818f6409f2404-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.068 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.068 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.068 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.068 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.068 DEBUG:boto:Method: GET 2024-03-04T16:27:41.069 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.069 DEBUG:boto:Data: 2024-03-04T16:27:41.069 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.069 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.069 DEBUG:boto:Params: {} 2024-03-04T16:27:41.069 DEBUG:boto:Token: None 2024-03-04T16:27:41.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043f56e29bc5260cc-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.072 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.073 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.073 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.073 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.073 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.073 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.073 DEBUG:boto:Data: 2024-03-04T16:27:41.073 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.073 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.073 DEBUG:boto:Params: {} 2024-03-04T16:27:41.074 DEBUG:boto:Token: None 2024-03-04T16:27:41.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HVlyxnrB4sCuvcaL1pUDX6uXd3w= 2024-03-04T16:27:41.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HVlyxnrB4sCuvcaL1pUDX6uXd3w=', 'Content-Length': '0'} 2024-03-04T16:27:41.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d1abe813937c5d9-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.076 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.076 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.076 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.076 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.076 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.077 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.077 DEBUG:boto:Data: 2024-03-04T16:27:41.077 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.077 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.077 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.077 DEBUG:boto:Params: {} 2024-03-04T16:27:41.077 DEBUG:boto:Token: None 2024-03-04T16:27:41.077 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.078 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HVlyxnrB4sCuvcaL1pUDX6uXd3w= 2024-03-04T16:27:41.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HVlyxnrB4sCuvcaL1pUDX6uXd3w=', 'Content-Length': '0'} 2024-03-04T16:27:41.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a0ebcfa72f92910-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:41.080 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.080 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.080 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.080 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.080 DEBUG:boto:Method: GET 2024-03-04T16:27:41.081 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.081 DEBUG:boto:Data: 2024-03-04T16:27:41.081 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.081 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.081 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.081 DEBUG:boto:Params: {} 2024-03-04T16:27:41.081 DEBUG:boto:Token: None 2024-03-04T16:27:41.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a55d2e9d7bccc8d6-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.083 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.084 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.084 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.084 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.084 DEBUG:boto:Method: GET 2024-03-04T16:27:41.084 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.084 DEBUG:boto:Data: 2024-03-04T16:27:41.084 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.084 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.084 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.084 DEBUG:boto:Params: {} 2024-03-04T16:27:41.085 DEBUG:boto:Token: None 2024-03-04T16:27:41.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c81826e632f9483-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.128 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.128 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.128 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.129 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.129 DEBUG:boto:Method: GET 2024-03-04T16:27:41.129 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.129 DEBUG:boto:Data: 2024-03-04T16:27:41.129 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.129 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.130 DEBUG:boto:Params: {} 2024-03-04T16:27:41.130 DEBUG:boto:Token: None 2024-03-04T16:27:41.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec41ae07e716a12-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.133 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.134 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.134 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.134 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.134 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.134 DEBUG:boto:Method: GET 2024-03-04T16:27:41.134 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.134 DEBUG:boto:Data: 2024-03-04T16:27:41.134 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.135 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.135 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.135 DEBUG:boto:Params: {} 2024-03-04T16:27:41.135 DEBUG:boto:Token: None 2024-03-04T16:27:41.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27c8debd4a49515-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.180 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.181 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.181 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.181 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.181 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.182 DEBUG:boto:Data: 2024-03-04T16:27:41.182 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.182 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.182 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.182 DEBUG:boto:Params: {} 2024-03-04T16:27:41.182 DEBUG:boto:Token: None 2024-03-04T16:27:41.182 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YqVdGPayUicEhHlIFEcv+lYfhyc= 2024-03-04T16:27:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YqVdGPayUicEhHlIFEcv+lYfhyc=', 'Content-Length': '0'} 2024-03-04T16:27:41.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073989cca025a6f34-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.188 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.188 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.188 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.188 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.189 DEBUG:boto:Data: 2024-03-04T16:27:41.189 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.189 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.189 DEBUG:boto:Params: {} 2024-03-04T16:27:41.189 DEBUG:boto:Token: None 2024-03-04T16:27:41.189 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YqVdGPayUicEhHlIFEcv+lYfhyc= 2024-03-04T16:27:41.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YqVdGPayUicEhHlIFEcv+lYfhyc=', 'Content-Length': '0'} 2024-03-04T16:27:41.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000925d83c22976bdc0-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:41.192 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.192 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.192 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.193 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.193 DEBUG:boto:Method: GET 2024-03-04T16:27:41.193 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.193 DEBUG:boto:Data: 2024-03-04T16:27:41.193 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.193 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.194 DEBUG:boto:Params: {} 2024-03-04T16:27:41.194 DEBUG:boto:Token: None 2024-03-04T16:27:41.194 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7d89007d651389d-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.196 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.196 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:41.196 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.197 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.197 DEBUG:boto:Method: GET 2024-03-04T16:27:41.197 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.197 DEBUG:boto:Data: 2024-03-04T16:27:41.197 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.197 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.197 DEBUG:boto:Params: {} 2024-03-04T16:27:41.197 DEBUG:boto:Token: None 2024-03-04T16:27:41.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:41.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo= 2024-03-04T16:27:41.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PhTYnSBM9oHupQXJ/dLxDZhNPVo=', 'Content-Length': '0'} 2024-03-04T16:27:41.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000013375412de06aab-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.240 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.240 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.241 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.241 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.241 DEBUG:boto:Method: GET 2024-03-04T16:27:41.241 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.241 DEBUG:boto:Data: 2024-03-04T16:27:41.241 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.242 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.242 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.242 DEBUG:boto:Params: {} 2024-03-04T16:27:41.242 DEBUG:boto:Token: None 2024-03-04T16:27:41.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a862a1782525ce98-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.246 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.246 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.246 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.246 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.246 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.246 DEBUG:boto:Method: GET 2024-03-04T16:27:41.246 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:41.247 DEBUG:boto:Data: 2024-03-04T16:27:41.247 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.247 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:41.247 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.247 DEBUG:boto:Params: {} 2024-03-04T16:27:41.247 DEBUG:boto:Token: None 2024-03-04T16:27:41.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.247 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04e6e1bf9079ea9-0065e5f67d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.251 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.251 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone3, test-zone1} 2024-03-04T16:27:41.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone3, test-zone2} 2024-03-04T16:27:41.251 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.251 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.251 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.251 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:41.252 DEBUG:boto:Data: 2024-03-04T16:27:41.252 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.252 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.252 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.252 DEBUG:boto:Params: {} 2024-03-04T16:27:41.252 DEBUG:boto:Token: None 2024-03-04T16:27:41.253 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.253 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/zKJyHVWNEWgTwnvIgYalPZ3DF4= 2024-03-04T16:27:41.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/zKJyHVWNEWgTwnvIgYalPZ3DF4=', 'Content-Length': '0'} 2024-03-04T16:27:41.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085b131835190ad8a-0065e5f67d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.255 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.255 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.255 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.256 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:41.256 DEBUG:boto:Data: 2024-03-04T16:27:41.256 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.256 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.256 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.256 DEBUG:boto:Params: {} 2024-03-04T16:27:41.256 DEBUG:boto:Token: None 2024-03-04T16:27:41.256 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.256 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/zKJyHVWNEWgTwnvIgYalPZ3DF4= 2024-03-04T16:27:41.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/zKJyHVWNEWgTwnvIgYalPZ3DF4=', 'Content-Length': '0'} 2024-03-04T16:27:41.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d45059927d345ac-0065e5f67d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:27:41.259 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.259 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.259 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.259 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.259 DEBUG:boto:Method: GET 2024-03-04T16:27:41.260 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.260 DEBUG:boto:Data: 2024-03-04T16:27:41.260 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.260 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.260 DEBUG:boto:Params: {} 2024-03-04T16:27:41.260 DEBUG:boto:Token: None 2024-03-04T16:27:41.260 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/?versions 2024-03-04T16:27:41.260 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U= 2024-03-04T16:27:41.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U=', 'Content-Length': '0'} 2024-03-04T16:27:41.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e633a237f6f34a73-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.263 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.264 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:41.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:41.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:41.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:41.264 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:27:41.264 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.264 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.264 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.264 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.264 DEBUG:boto:Method: GET 2024-03-04T16:27:41.264 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.265 DEBUG:boto:Data: 2024-03-04T16:27:41.265 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.265 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.265 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.265 DEBUG:boto:Params: {} 2024-03-04T16:27:41.265 DEBUG:boto:Token: None 2024-03-04T16:27:41.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/?versions 2024-03-04T16:27:41.265 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U= 2024-03-04T16:27:41.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U=', 'Content-Length': '0'} 2024-03-04T16:27:41.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1724ace2778a9b-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.268 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.268 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:27:41.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:27:41.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:27:41.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:27:41.269 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.269 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.269 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.269 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.269 DEBUG:boto:Method: GET 2024-03-04T16:27:41.269 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.269 DEBUG:boto:Data: 2024-03-04T16:27:41.269 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.270 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.270 DEBUG:boto:Params: {} 2024-03-04T16:27:41.270 DEBUG:boto:Token: None 2024-03-04T16:27:41.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/?versions 2024-03-04T16:27:41.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U= 2024-03-04T16:27:41.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U=', 'Content-Length': '0'} 2024-03-04T16:27:41.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149e15b75acf8457-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.273 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.273 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.273 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.273 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.273 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.273 DEBUG:boto:Method: GET 2024-03-04T16:27:41.273 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:27:41.274 DEBUG:boto:Data: 2024-03-04T16:27:41.274 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.274 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.274 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.274 DEBUG:boto:Params: {} 2024-03-04T16:27:41.274 DEBUG:boto:Token: None 2024-03-04T16:27:41.274 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/?versions 2024-03-04T16:27:41.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U= 2024-03-04T16:27:41.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2aiSW+e9Yc9jZgiVkLuiSa+A89U=', 'Content-Length': '0'} 2024-03-04T16:27:41.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000676a28293740980-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.324 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:41.325 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.325 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.325 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.325 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.325 DEBUG:boto:Method: GET 2024-03-04T16:27:41.325 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.325 DEBUG:boto:Data: 2024-03-04T16:27:41.325 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.326 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.326 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.326 DEBUG:boto:Params: {} 2024-03-04T16:27:41.326 DEBUG:boto:Token: None 2024-03-04T16:27:41.326 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aeb123e9a4979e5f-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.329 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.329 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.329 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.329 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.329 DEBUG:boto:Method: GET 2024-03-04T16:27:41.329 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.329 DEBUG:boto:Data: 2024-03-04T16:27:41.329 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.329 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.330 DEBUG:boto:Params: {} 2024-03-04T16:27:41.330 DEBUG:boto:Token: None 2024-03-04T16:27:41.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b443ffec67a0c5f5-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.332 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.332 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.332 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.332 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.332 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.332 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.333 DEBUG:boto:Data: 2024-03-04T16:27:41.333 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.333 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.333 DEBUG:boto:Params: {} 2024-03-04T16:27:41.333 DEBUG:boto:Token: None 2024-03-04T16:27:41.333 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PwFluHuv5Pi9wnxloNxL9zJL04o= 2024-03-04T16:27:41.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PwFluHuv5Pi9wnxloNxL9zJL04o=', 'Content-Length': '0'} 2024-03-04T16:27:41.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000717e746984215fa7-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.335 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.336 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.336 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.336 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.336 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.336 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.336 DEBUG:boto:Data: 2024-03-04T16:27:41.336 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.336 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.336 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.336 DEBUG:boto:Params: {} 2024-03-04T16:27:41.336 DEBUG:boto:Token: None 2024-03-04T16:27:41.337 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.337 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PwFluHuv5Pi9wnxloNxL9zJL04o= 2024-03-04T16:27:41.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PwFluHuv5Pi9wnxloNxL9zJL04o=', 'Content-Length': '0'} 2024-03-04T16:27:41.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a4469f7fa02e30e-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:41.339 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.339 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.339 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.339 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.339 DEBUG:boto:Method: GET 2024-03-04T16:27:41.339 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.339 DEBUG:boto:Data: 2024-03-04T16:27:41.339 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.339 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.340 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.340 DEBUG:boto:Params: {} 2024-03-04T16:27:41.340 DEBUG:boto:Token: None 2024-03-04T16:27:41.340 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.340 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080abe4c327c238e4-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.342 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.342 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.342 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.342 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.343 DEBUG:boto:Method: GET 2024-03-04T16:27:41.343 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.343 DEBUG:boto:Data: 2024-03-04T16:27:41.343 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.343 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.343 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.343 DEBUG:boto:Params: {} 2024-03-04T16:27:41.343 DEBUG:boto:Token: None 2024-03-04T16:27:41.343 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.343 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b457bd8e075f3b38-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.346 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.346 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.346 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.346 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.346 DEBUG:boto:Method: GET 2024-03-04T16:27:41.346 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.346 DEBUG:boto:Data: 2024-03-04T16:27:41.346 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.346 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.346 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.346 DEBUG:boto:Params: {} 2024-03-04T16:27:41.347 DEBUG:boto:Token: None 2024-03-04T16:27:41.347 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.347 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a21126439f8c4d70-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.350 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.350 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.350 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.350 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.350 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.350 DEBUG:boto:Method: GET 2024-03-04T16:27:41.350 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.350 DEBUG:boto:Data: 2024-03-04T16:27:41.351 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.351 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.351 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.351 DEBUG:boto:Params: {} 2024-03-04T16:27:41.351 DEBUG:boto:Token: None 2024-03-04T16:27:41.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.351 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060a591de29826f8c-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.354 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.354 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.355 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.355 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.355 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:41.355 DEBUG:boto:Data: 2024-03-04T16:27:41.355 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.355 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.355 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.355 DEBUG:boto:Params: {} 2024-03-04T16:27:41.355 DEBUG:boto:Token: None 2024-03-04T16:27:41.355 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy 2024-03-04T16:27:41.355 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:50e0k8y+rQj6KLSD2w+2fJUjSsQ= 2024-03-04T16:27:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:50e0k8y+rQj6KLSD2w+2fJUjSsQ=', 'Content-Length': '0'} 2024-03-04T16:27:41.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d22ddc6fb9127074-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.358 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.358 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.358 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.358 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:27:41.358 DEBUG:boto:Data: 2024-03-04T16:27:41.358 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.358 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.359 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.359 DEBUG:boto:Params: {} 2024-03-04T16:27:41.359 DEBUG:boto:Token: None 2024-03-04T16:27:41.359 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy 2024-03-04T16:27:41.359 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:50e0k8y+rQj6KLSD2w+2fJUjSsQ= 2024-03-04T16:27:41.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:50e0k8y+rQj6KLSD2w+2fJUjSsQ=', 'Content-Length': '0'} 2024-03-04T16:27:41.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7b550fe22df12d1-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:27:41.362 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.362 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.362 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.362 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.362 DEBUG:boto:Method: GET 2024-03-04T16:27:41.362 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.362 DEBUG:boto:Data: 2024-03-04T16:27:41.362 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.362 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.362 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.362 DEBUG:boto:Params: {} 2024-03-04T16:27:41.363 DEBUG:boto:Token: None 2024-03-04T16:27:41.363 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.363 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000883500c57eb9a0d8-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.365 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:27:41.365 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:27:41.365 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.365 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.365 DEBUG:boto:Method: GET 2024-03-04T16:27:41.365 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.365 DEBUG:boto:Data: 2024-03-04T16:27:41.365 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.366 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.366 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.366 DEBUG:boto:Params: {} 2024-03-04T16:27:41.366 DEBUG:boto:Token: None 2024-03-04T16:27:41.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:27:41.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY= 2024-03-04T16:27:41.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhYmXEPQKWvtyZtRRZJxhZRYjpY=', 'Content-Length': '0'} 2024-03-04T16:27:41.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062ac7945c7eb4c0b-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.368 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.368 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.369 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.369 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.369 DEBUG:boto:Method: GET 2024-03-04T16:27:41.369 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.369 DEBUG:boto:Data: 2024-03-04T16:27:41.369 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.369 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.369 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.369 DEBUG:boto:Params: {} 2024-03-04T16:27:41.369 DEBUG:boto:Token: None 2024-03-04T16:27:41.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c62631f6d2b1a8d-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.420 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.420 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.421 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.421 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.421 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.421 DEBUG:boto:Method: GET 2024-03-04T16:27:41.421 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:27:41.421 DEBUG:boto:Data: 2024-03-04T16:27:41.421 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.421 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.421 DEBUG:boto:Params: {} 2024-03-04T16:27:41.422 DEBUG:boto:Token: None 2024-03-04T16:27:41.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7600e99093f3c19-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.425 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:41.426 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.426 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.426 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.426 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.426 DEBUG:boto:Method: GET 2024-03-04T16:27:41.426 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.426 DEBUG:boto:Data: 2024-03-04T16:27:41.427 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.427 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.427 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.427 DEBUG:boto:Params: {} 2024-03-04T16:27:41.427 DEBUG:boto:Token: None 2024-03-04T16:27:41.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.427 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076c4c1e6841ac232-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.430 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.430 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.430 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.430 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.430 DEBUG:boto:Method: GET 2024-03-04T16:27:41.430 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.431 DEBUG:boto:Data: 2024-03-04T16:27:41.431 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.431 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.431 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.431 DEBUG:boto:Params: {} 2024-03-04T16:27:41.431 DEBUG:boto:Token: None 2024-03-04T16:27:41.431 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004035ce9518453e87-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.480 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.480 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.481 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.481 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.481 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.481 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.481 DEBUG:boto:Data: 2024-03-04T16:27:41.481 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.481 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.481 DEBUG:boto:Params: {} 2024-03-04T16:27:41.481 DEBUG:boto:Token: None 2024-03-04T16:27:41.481 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lBH+BjDRgVvXgm6qXV6Q8/l/OJA= 2024-03-04T16:27:41.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lBH+BjDRgVvXgm6qXV6Q8/l/OJA=', 'Content-Length': '0'} 2024-03-04T16:27:41.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005c60885019acd04-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.484 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.485 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.485 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.485 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.485 DEBUG:boto:Data: 2024-03-04T16:27:41.485 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.485 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.485 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.485 DEBUG:boto:Params: {} 2024-03-04T16:27:41.485 DEBUG:boto:Token: None 2024-03-04T16:27:41.486 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lBH+BjDRgVvXgm6qXV6Q8/l/OJA= 2024-03-04T16:27:41.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lBH+BjDRgVvXgm6qXV6Q8/l/OJA=', 'Content-Length': '0'} 2024-03-04T16:27:41.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a27311f7d779dab-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:41.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.488 DEBUG:boto:Method: GET 2024-03-04T16:27:41.488 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.488 DEBUG:boto:Data: 2024-03-04T16:27:41.488 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.488 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.488 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.489 DEBUG:boto:Params: {} 2024-03-04T16:27:41.489 DEBUG:boto:Token: None 2024-03-04T16:27:41.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000309553c2750772da-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.536 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.536 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.536 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.536 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.537 DEBUG:boto:Method: GET 2024-03-04T16:27:41.537 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.537 DEBUG:boto:Data: 2024-03-04T16:27:41.537 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.537 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.537 DEBUG:boto:Params: {} 2024-03-04T16:27:41.537 DEBUG:boto:Token: None 2024-03-04T16:27:41.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bc329cb44ce49fc-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.581 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.582 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.582 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.582 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.582 DEBUG:boto:Method: GET 2024-03-04T16:27:41.582 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.583 DEBUG:boto:Data: 2024-03-04T16:27:41.583 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.583 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.583 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.583 DEBUG:boto:Params: {} 2024-03-04T16:27:41.584 DEBUG:boto:Token: None 2024-03-04T16:27:41.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776ab3111e7b38a5-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.588 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.588 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.589 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.589 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.589 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.589 DEBUG:boto:Method: GET 2024-03-04T16:27:41.589 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.590 DEBUG:boto:Data: 2024-03-04T16:27:41.590 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.590 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.590 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.590 DEBUG:boto:Params: {} 2024-03-04T16:27:41.591 DEBUG:boto:Token: None 2024-03-04T16:27:41.591 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.591 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c7e6f812f9d47e0-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.595 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.595 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.596 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.596 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.596 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.596 DEBUG:boto:Data: 2024-03-04T16:27:41.596 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.596 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.597 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.597 DEBUG:boto:Params: {} 2024-03-04T16:27:41.597 DEBUG:boto:Token: None 2024-03-04T16:27:41.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hxeb1OrgB8ayEHpNgx+okQlqP00= 2024-03-04T16:27:41.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hxeb1OrgB8ayEHpNgx+okQlqP00=', 'Content-Length': '0'} 2024-03-04T16:27:41.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041f360661aaf59ba-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.600 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.601 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.601 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.601 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.601 DEBUG:boto:Data: 2024-03-04T16:27:41.601 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.602 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.602 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.602 DEBUG:boto:Params: {} 2024-03-04T16:27:41.602 DEBUG:boto:Token: None 2024-03-04T16:27:41.602 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.603 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hxeb1OrgB8ayEHpNgx+okQlqP00= 2024-03-04T16:27:41.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hxeb1OrgB8ayEHpNgx+okQlqP00=', 'Content-Length': '0'} 2024-03-04T16:27:41.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075a14ed60bfa0cc2-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:27:41.606 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.606 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.606 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.606 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.606 DEBUG:boto:Method: GET 2024-03-04T16:27:41.607 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.607 DEBUG:boto:Data: 2024-03-04T16:27:41.607 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.607 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.607 DEBUG:boto:Params: {} 2024-03-04T16:27:41.607 DEBUG:boto:Token: None 2024-03-04T16:27:41.608 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086568caa1f01fc7a-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.652 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.652 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:41.652 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.652 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.653 DEBUG:boto:Method: GET 2024-03-04T16:27:41.653 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.653 DEBUG:boto:Data: 2024-03-04T16:27:41.653 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.653 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.653 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.653 DEBUG:boto:Params: {} 2024-03-04T16:27:41.653 DEBUG:boto:Token: None 2024-03-04T16:27:41.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:27:41.653 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0= 2024-03-04T16:27:41.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rPOn9bfWXhqvJv1RAzEooP5AVw0=', 'Content-Length': '0'} 2024-03-04T16:27:41.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004484c5a6bf86a281-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.696 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.696 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.696 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.696 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.697 DEBUG:boto:Method: GET 2024-03-04T16:27:41.697 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.697 DEBUG:boto:Data: 2024-03-04T16:27:41.697 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.697 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.697 DEBUG:boto:Params: {} 2024-03-04T16:27:41.697 DEBUG:boto:Token: None 2024-03-04T16:27:41.697 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f33fd2effc36e4-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.744 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.745 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.745 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.745 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.745 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.745 DEBUG:boto:Method: GET 2024-03-04T16:27:41.745 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:27:41.745 DEBUG:boto:Data: 2024-03-04T16:27:41.745 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.745 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.745 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.745 DEBUG:boto:Params: {} 2024-03-04T16:27:41.746 DEBUG:boto:Token: None 2024-03-04T16:27:41.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019a632b775feeb8-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.749 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:41.749 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.750 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.750 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.750 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.750 DEBUG:boto:Method: GET 2024-03-04T16:27:41.750 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.750 DEBUG:boto:Data: 2024-03-04T16:27:41.750 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.750 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.750 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.750 DEBUG:boto:Params: {} 2024-03-04T16:27:41.751 DEBUG:boto:Token: None 2024-03-04T16:27:41.751 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.751 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096de4f4ededac021-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.796 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.796 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.796 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.797 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.797 DEBUG:boto:Method: GET 2024-03-04T16:27:41.797 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.797 DEBUG:boto:Data: 2024-03-04T16:27:41.797 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.797 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.797 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.797 DEBUG:boto:Params: {} 2024-03-04T16:27:41.797 DEBUG:boto:Token: None 2024-03-04T16:27:41.797 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ab18771efa1d02a-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.800 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.800 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.800 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.800 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.800 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.801 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.801 DEBUG:boto:Data: 2024-03-04T16:27:41.801 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.801 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.801 DEBUG:boto:Params: {} 2024-03-04T16:27:41.801 DEBUG:boto:Token: None 2024-03-04T16:27:41.801 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.801 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DJBPWzHLFNvUzkSBJfIg6X+XEmA= 2024-03-04T16:27:41.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DJBPWzHLFNvUzkSBJfIg6X+XEmA=', 'Content-Length': '0'} 2024-03-04T16:27:41.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f2b5e0333ad0a9-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.804 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.804 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.804 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.804 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.804 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.804 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.804 DEBUG:boto:Data: 2024-03-04T16:27:41.804 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.804 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.804 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.805 DEBUG:boto:Params: {} 2024-03-04T16:27:41.805 DEBUG:boto:Token: None 2024-03-04T16:27:41.805 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DJBPWzHLFNvUzkSBJfIg6X+XEmA= 2024-03-04T16:27:41.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DJBPWzHLFNvUzkSBJfIg6X+XEmA=', 'Content-Length': '0'} 2024-03-04T16:27:41.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba5ad33dd85c9643-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:41.808 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.808 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.808 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.808 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.808 DEBUG:boto:Method: GET 2024-03-04T16:27:41.808 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.808 DEBUG:boto:Data: 2024-03-04T16:27:41.808 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.808 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.809 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.809 DEBUG:boto:Params: {} 2024-03-04T16:27:41.809 DEBUG:boto:Token: None 2024-03-04T16:27:41.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059a4fd545754da79-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.856 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.856 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.856 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.856 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.857 DEBUG:boto:Method: GET 2024-03-04T16:27:41.857 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.857 DEBUG:boto:Data: 2024-03-04T16:27:41.857 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.857 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.857 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.857 DEBUG:boto:Params: {} 2024-03-04T16:27:41.858 DEBUG:boto:Token: None 2024-03-04T16:27:41.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.858 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006924e04f4372f7ed-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.860 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.861 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.861 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.861 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.861 DEBUG:boto:Method: GET 2024-03-04T16:27:41.861 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.861 DEBUG:boto:Data: 2024-03-04T16:27:41.861 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.861 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.862 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.862 DEBUG:boto:Params: {} 2024-03-04T16:27:41.862 DEBUG:boto:Token: None 2024-03-04T16:27:41.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1d50aad8c0f35f6-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.908 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.909 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.909 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.909 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.909 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.909 DEBUG:boto:Method: GET 2024-03-04T16:27:41.909 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.909 DEBUG:boto:Data: 2024-03-04T16:27:41.909 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.909 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.909 DEBUG:boto:Params: {} 2024-03-04T16:27:41.910 DEBUG:boto:Token: None 2024-03-04T16:27:41.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb79583bc953d58-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.913 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:41.914 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.914 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.914 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.914 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.914 DEBUG:boto:Data: 2024-03-04T16:27:41.914 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.914 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.914 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.914 DEBUG:boto:Params: {} 2024-03-04T16:27:41.914 DEBUG:boto:Token: None 2024-03-04T16:27:41.915 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.915 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o50yTdaDsYrebrgXtMZ2nXl/evk= 2024-03-04T16:27:41.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o50yTdaDsYrebrgXtMZ2nXl/evk=', 'Content-Length': '0'} 2024-03-04T16:27:41.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f815adeb4e47992e-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.918 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.918 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.918 DEBUG:boto:Method: HEAD 2024-03-04T16:27:41.918 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.918 DEBUG:boto:Data: 2024-03-04T16:27:41.918 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.918 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.918 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.919 DEBUG:boto:Params: {} 2024-03-04T16:27:41.919 DEBUG:boto:Token: None 2024-03-04T16:27:41.919 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.919 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o50yTdaDsYrebrgXtMZ2nXl/evk= 2024-03-04T16:27:41.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o50yTdaDsYrebrgXtMZ2nXl/evk=', 'Content-Length': '0'} 2024-03-04T16:27:41.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccae6055e5b73d4c-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:27:41.922 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.922 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.922 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.922 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.923 DEBUG:boto:Method: GET 2024-03-04T16:27:41.923 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.923 DEBUG:boto:Data: 2024-03-04T16:27:41.923 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.923 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.923 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.923 DEBUG:boto:Params: {} 2024-03-04T16:27:41.923 DEBUG:boto:Token: None 2024-03-04T16:27:41.924 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.924 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a5bf4e62ce92801-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.972 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.972 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:41.973 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.973 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.973 DEBUG:boto:Method: GET 2024-03-04T16:27:41.973 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.973 DEBUG:boto:Data: 2024-03-04T16:27:41.973 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:41.973 DEBUG:boto:Port: 8000 2024-03-04T16:27:41.973 DEBUG:boto:Params: {} 2024-03-04T16:27:41.973 DEBUG:boto:Token: None 2024-03-04T16:27:41.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:27:41.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8= 2024-03-04T16:27:41.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jdE4gVAhPLCm8ip+KpaZKFLlhR8=', 'Content-Length': '0'} 2024-03-04T16:27:41.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037efc4375cb7db37-0065e5f67d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:41.976 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:41.976 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:41.977 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.977 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.977 DEBUG:boto:Method: GET 2024-03-04T16:27:41.977 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:41.977 DEBUG:boto:Data: 2024-03-04T16:27:41.977 DEBUG:boto:Headers: {} 2024-03-04T16:27:41.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:41.978 DEBUG:boto:Port: 8001 2024-03-04T16:27:41.978 DEBUG:boto:Params: {} 2024-03-04T16:27:41.978 DEBUG:boto:Token: None 2024-03-04T16:27:41.978 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:41 GMT /sbnvju-21/ 2024-03-04T16:27:41.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE= 2024-03-04T16:27:41.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE9KDpwRMyFHxScEsJyCNhNTMsE=', 'Content-Length': '0'} 2024-03-04T16:27:41.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06f0b4db2e49ea7-0065e5f67d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.024 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.025 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.025 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.025 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:42.025 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:42.025 DEBUG:boto:Method: GET 2024-03-04T16:27:42.025 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:27:42.025 DEBUG:boto:Data: 2024-03-04T16:27:42.026 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.026 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.026 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.026 DEBUG:boto:Params: {} 2024-03-04T16:27:42.026 DEBUG:boto:Token: None 2024-03-04T16:27:42.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.026 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8= 2024-03-04T16:27:42.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8=', 'Content-Length': '0'} 2024-03-04T16:27:42.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000831aef0b2e1e0c33-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.029 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:42.030 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.030 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.030 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.030 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.030 DEBUG:boto:Method: GET 2024-03-04T16:27:42.030 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.031 DEBUG:boto:Data: 2024-03-04T16:27:42.031 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.031 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.031 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.031 DEBUG:boto:Params: {} 2024-03-04T16:27:42.031 DEBUG:boto:Token: None 2024-03-04T16:27:42.031 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.032 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005054858cf8c71ac0-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.034 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.034 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.034 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.034 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.034 DEBUG:boto:Method: GET 2024-03-04T16:27:42.034 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.034 DEBUG:boto:Data: 2024-03-04T16:27:42.034 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.034 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.035 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.035 DEBUG:boto:Params: {} 2024-03-04T16:27:42.035 DEBUG:boto:Token: None 2024-03-04T16:27:42.035 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.035 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d83e74c677d5120-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.084 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.084 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.084 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.084 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.085 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.085 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.085 DEBUG:boto:Data: 2024-03-04T16:27:42.085 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.085 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.085 DEBUG:boto:Params: {} 2024-03-04T16:27:42.085 DEBUG:boto:Token: None 2024-03-04T16:27:42.086 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.086 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VgCvfUms1BY/Im1X3es2I+LxrWU= 2024-03-04T16:27:42.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VgCvfUms1BY/Im1X3es2I+LxrWU=', 'Content-Length': '0'} 2024-03-04T16:27:42.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e4048e1e3e28112-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.088 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.089 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.089 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.089 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.089 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.090 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.090 DEBUG:boto:Data: 2024-03-04T16:27:42.091 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.091 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.091 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.091 DEBUG:boto:Params: {} 2024-03-04T16:27:42.092 DEBUG:boto:Token: None 2024-03-04T16:27:42.092 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.092 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VgCvfUms1BY/Im1X3es2I+LxrWU= 2024-03-04T16:27:42.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VgCvfUms1BY/Im1X3es2I+LxrWU=', 'Content-Length': '0'} 2024-03-04T16:27:42.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6d33d7cf18a8c72-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:42.094 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.094 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.094 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.095 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.095 DEBUG:boto:Method: GET 2024-03-04T16:27:42.095 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.095 DEBUG:boto:Data: 2024-03-04T16:27:42.095 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.095 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.095 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.095 DEBUG:boto:Params: {} 2024-03-04T16:27:42.095 DEBUG:boto:Token: None 2024-03-04T16:27:42.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.096 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2bbec46943dc886-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.098 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.098 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.098 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.098 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.098 DEBUG:boto:Method: GET 2024-03-04T16:27:42.098 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.098 DEBUG:boto:Data: 2024-03-04T16:27:42.098 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.099 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.099 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.099 DEBUG:boto:Params: {} 2024-03-04T16:27:42.099 DEBUG:boto:Token: None 2024-03-04T16:27:42.099 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.099 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000caff52b85525b38b-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.144 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.144 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.144 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.144 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.145 DEBUG:boto:Method: GET 2024-03-04T16:27:42.145 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.145 DEBUG:boto:Data: 2024-03-04T16:27:42.145 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.145 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.145 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.145 DEBUG:boto:Params: {} 2024-03-04T16:27:42.145 DEBUG:boto:Token: None 2024-03-04T16:27:42.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.146 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8= 2024-03-04T16:27:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8=', 'Content-Length': '0'} 2024-03-04T16:27:42.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095df2e140719f30d-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.148 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.149 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.149 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.149 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.149 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.149 DEBUG:boto:Method: GET 2024-03-04T16:27:42.149 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.149 DEBUG:boto:Data: 2024-03-04T16:27:42.149 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.150 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.150 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.150 DEBUG:boto:Params: {} 2024-03-04T16:27:42.150 DEBUG:boto:Token: None 2024-03-04T16:27:42.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8= 2024-03-04T16:27:42.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8=', 'Content-Length': '0'} 2024-03-04T16:27:42.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cf1803e7270737f-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.153 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.153 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.153 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.153 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.153 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.154 DEBUG:boto:Data: 2024-03-04T16:27:42.154 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.154 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.154 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.154 DEBUG:boto:Params: {} 2024-03-04T16:27:42.154 DEBUG:boto:Token: None 2024-03-04T16:27:42.154 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NS1sIj99uwjAPLm2g6UsvxO53WQ= 2024-03-04T16:27:42.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NS1sIj99uwjAPLm2g6UsvxO53WQ=', 'Content-Length': '0'} 2024-03-04T16:27:42.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f420e1a2e4fb6eaa-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.158 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.158 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.158 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.158 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.158 DEBUG:boto:Data: 2024-03-04T16:27:42.158 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.158 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.158 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.158 DEBUG:boto:Params: {} 2024-03-04T16:27:42.159 DEBUG:boto:Token: None 2024-03-04T16:27:42.159 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NS1sIj99uwjAPLm2g6UsvxO53WQ= 2024-03-04T16:27:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NS1sIj99uwjAPLm2g6UsvxO53WQ=', 'Content-Length': '0'} 2024-03-04T16:27:42.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005175d456d7b9404d-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:27:42.162 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.162 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.162 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.162 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.162 DEBUG:boto:Method: GET 2024-03-04T16:27:42.162 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.162 DEBUG:boto:Data: 2024-03-04T16:27:42.162 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.163 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.163 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.163 DEBUG:boto:Params: {} 2024-03-04T16:27:42.163 DEBUG:boto:Token: None 2024-03-04T16:27:42.163 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.163 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f43526704a0c7aa6-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.208 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.208 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.208 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.208 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.209 DEBUG:boto:Method: GET 2024-03-04T16:27:42.209 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.209 DEBUG:boto:Data: 2024-03-04T16:27:42.209 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.209 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.209 DEBUG:boto:Params: {} 2024-03-04T16:27:42.209 DEBUG:boto:Token: None 2024-03-04T16:27:42.209 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:27:42.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw= 2024-03-04T16:27:42.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wEc2Izz47oDUp6fXINyqk523+Lw=', 'Content-Length': '0'} 2024-03-04T16:27:42.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000762cb8f68b955c36-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.253 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.253 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.253 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.253 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.253 DEBUG:boto:Method: GET 2024-03-04T16:27:42.253 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.254 DEBUG:boto:Data: 2024-03-04T16:27:42.254 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.254 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.254 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.254 DEBUG:boto:Params: {} 2024-03-04T16:27:42.254 DEBUG:boto:Token: None 2024-03-04T16:27:42.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8= 2024-03-04T16:27:42.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8=', 'Content-Length': '0'} 2024-03-04T16:27:42.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a515d36863800a95-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.257 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.257 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.258 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.258 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.258 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.258 DEBUG:boto:Method: GET 2024-03-04T16:27:42.258 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:27:42.258 DEBUG:boto:Data: 2024-03-04T16:27:42.258 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.258 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.258 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.258 DEBUG:boto:Params: {} 2024-03-04T16:27:42.258 DEBUG:boto:Token: None 2024-03-04T16:27:42.259 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.259 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8= 2024-03-04T16:27:42.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HEA6lfnmO6aO9s9/3eNvNG5EoM8=', 'Content-Length': '0'} 2024-03-04T16:27:42.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001903877a89593437-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.261 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone3, test-zone2} 2024-03-04T16:27:42.262 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.262 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.262 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.262 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:27:42.262 DEBUG:boto:Data: 2024-03-04T16:27:42.262 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.262 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.262 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.263 DEBUG:boto:Params: {} 2024-03-04T16:27:42.263 DEBUG:boto:Token: None 2024-03-04T16:27:42.263 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/ 2024-03-04T16:27:42.263 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8= 2024-03-04T16:27:42.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8=', 'Content-Length': '0'} 2024-03-04T16:27:42.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000850bfc3b978bafb6-0065e5f67e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.266 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.266 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.266 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.266 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.266 DEBUG:boto:Method: GET 2024-03-04T16:27:42.266 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.266 DEBUG:boto:Data: 2024-03-04T16:27:42.266 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.266 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.266 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.266 DEBUG:boto:Params: {} 2024-03-04T16:27:42.267 DEBUG:boto:Token: None 2024-03-04T16:27:42.267 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/?versions 2024-03-04T16:27:42.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE= 2024-03-04T16:27:42.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE=', 'Content-Length': '0'} 2024-03-04T16:27:42.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ec7b49e37650ce-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.312 DEBUG:boto:b'sbnvju-19obj-test-zone1999falseobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.313 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:27:42.313 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:27:42.313 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?acl 2024-03-04T16:27:42.313 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?acl 2024-03-04T16:27:42.313 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.313 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?acl 2024-03-04T16:27:42.313 DEBUG:boto:Data: 2024-03-04T16:27:42.313 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.313 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.314 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.314 DEBUG:boto:Params: {} 2024-03-04T16:27:42.314 DEBUG:boto:Token: None 2024-03-04T16:27:42.314 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-19/obj-test-zone1?acl 2024-03-04T16:27:42.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u6SEbgqdugYrxZW5QG00lMJNhdY= 2024-03-04T16:27:42.314 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u6SEbgqdugYrxZW5QG00lMJNhdY=', 'Content-Length': '0'} 2024-03-04T16:27:42.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016e7f5145176ae92-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.321 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.321 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.322 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.322 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:27:42.322 DEBUG:boto:Data: 2024-03-04T16:27:42.322 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.322 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.322 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.322 DEBUG:boto:Params: {} 2024-03-04T16:27:42.322 DEBUG:boto:Token: None 2024-03-04T16:27:42.322 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/ 2024-03-04T16:27:42.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8= 2024-03-04T16:27:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8=', 'Content-Length': '0'} 2024-03-04T16:27:42.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000080aa73f46a9df1b-0065e5f67e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.325 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.325 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.325 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.326 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.326 DEBUG:boto:Method: GET 2024-03-04T16:27:42.326 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.327 DEBUG:boto:Data: 2024-03-04T16:27:42.327 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.327 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.327 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.327 DEBUG:boto:Params: {} 2024-03-04T16:27:42.327 DEBUG:boto:Token: None 2024-03-04T16:27:42.327 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/?versions 2024-03-04T16:27:42.328 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE= 2024-03-04T16:27:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE=', 'Content-Length': '0'} 2024-03-04T16:27:42.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f25ed15e970268-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.330 DEBUG:boto:b'sbnvju-19obj-test-zone2999falseobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.331 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:27:42.331 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:27:42.331 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?acl 2024-03-04T16:27:42.331 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?acl 2024-03-04T16:27:42.331 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.331 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?acl 2024-03-04T16:27:42.331 DEBUG:boto:Data: 2024-03-04T16:27:42.331 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.331 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.331 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.331 DEBUG:boto:Params: {} 2024-03-04T16:27:42.332 DEBUG:boto:Token: None 2024-03-04T16:27:42.332 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-19/obj-test-zone2?acl 2024-03-04T16:27:42.332 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UAOsVnUeDvmeafYHYWHLbI5EgmM= 2024-03-04T16:27:42.332 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UAOsVnUeDvmeafYHYWHLbI5EgmM=', 'Content-Length': '0'} 2024-03-04T16:27:42.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000857e0e01179ce3c2-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.339 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.339 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.339 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.340 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:27:42.340 DEBUG:boto:Data: 2024-03-04T16:27:42.340 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.340 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.340 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.340 DEBUG:boto:Params: {} 2024-03-04T16:27:42.340 DEBUG:boto:Token: None 2024-03-04T16:27:42.340 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/ 2024-03-04T16:27:42.340 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8= 2024-03-04T16:27:42.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/XBYf9xuRR+b1ttO+8IusLPw/J8=', 'Content-Length': '0'} 2024-03-04T16:27:42.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec76709b75dfe422-0065e5f67e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.343 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:27:42.343 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:27:42.343 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.343 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.343 DEBUG:boto:Method: GET 2024-03-04T16:27:42.344 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.344 DEBUG:boto:Data: 2024-03-04T16:27:42.344 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.344 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.344 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.344 DEBUG:boto:Params: {} 2024-03-04T16:27:42.344 DEBUG:boto:Token: None 2024-03-04T16:27:42.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-19/?versions 2024-03-04T16:27:42.344 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE= 2024-03-04T16:27:42.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxFXVBJ4p+Q0X75iWD1sJ9TI5ZE=', 'Content-Length': '0'} 2024-03-04T16:27:42.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f79378778749c920-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.347 DEBUG:boto:b'sbnvju-19obj-test-zone3999falseobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.347 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:27:42.347 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:27:42.347 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?acl 2024-03-04T16:27:42.348 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?acl 2024-03-04T16:27:42.348 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.348 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?acl 2024-03-04T16:27:42.348 DEBUG:boto:Data: 2024-03-04T16:27:42.348 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.348 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.348 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.348 DEBUG:boto:Params: {} 2024-03-04T16:27:42.348 DEBUG:boto:Token: None 2024-03-04T16:27:42.348 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-19/obj-test-zone3?acl 2024-03-04T16:27:42.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PdmIv3dbF9sYEja9De+ZG8IImsA= 2024-03-04T16:27:42.349 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PdmIv3dbF9sYEja9De+ZG8IImsA=', 'Content-Length': '0'} 2024-03-04T16:27:42.355 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000408d97c69d99bbb0-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.355 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.355 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.355 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.356 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:27:42.356 DEBUG:boto:Data: 2024-03-04T16:27:42.356 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.356 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.356 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.356 DEBUG:boto:Params: {} 2024-03-04T16:27:42.356 DEBUG:boto:Token: None 2024-03-04T16:27:42.356 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/ 2024-03-04T16:27:42.356 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4= 2024-03-04T16:27:42.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4=', 'Content-Length': '0'} 2024-03-04T16:27:42.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8e0b8dbcf9182f4-0065e5f67e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.359 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.359 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.359 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.359 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.359 DEBUG:boto:Method: GET 2024-03-04T16:27:42.359 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.359 DEBUG:boto:Data: 2024-03-04T16:27:42.359 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.360 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.360 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.360 DEBUG:boto:Params: {} 2024-03-04T16:27:42.360 DEBUG:boto:Token: None 2024-03-04T16:27:42.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/?versions 2024-03-04T16:27:42.360 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc= 2024-03-04T16:27:42.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc=', 'Content-Length': '0'} 2024-03-04T16:27:42.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003614886023b94684-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.363 DEBUG:boto:b'sbnvju-20obj-test-zone1999falseobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.364 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:27:42.364 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:27:42.364 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?acl 2024-03-04T16:27:42.364 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?acl 2024-03-04T16:27:42.364 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.364 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?acl 2024-03-04T16:27:42.364 DEBUG:boto:Data: 2024-03-04T16:27:42.365 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.365 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.365 DEBUG:boto:Params: {} 2024-03-04T16:27:42.365 DEBUG:boto:Token: None 2024-03-04T16:27:42.365 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-20/obj-test-zone1?acl 2024-03-04T16:27:42.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/aGXeVmGJOmr/EGcNsnIAbiRb14= 2024-03-04T16:27:42.365 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/aGXeVmGJOmr/EGcNsnIAbiRb14=', 'Content-Length': '0'} 2024-03-04T16:27:42.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b45c41f5588964b-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.371 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.371 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.371 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.371 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:27:42.372 DEBUG:boto:Data: 2024-03-04T16:27:42.372 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.372 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.372 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.372 DEBUG:boto:Params: {} 2024-03-04T16:27:42.372 DEBUG:boto:Token: None 2024-03-04T16:27:42.372 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/ 2024-03-04T16:27:42.372 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4= 2024-03-04T16:27:42.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4=', 'Content-Length': '0'} 2024-03-04T16:27:42.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7f3d19f10c2528c-0065e5f67e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.375 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.375 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.375 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.375 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.375 DEBUG:boto:Method: GET 2024-03-04T16:27:42.375 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.376 DEBUG:boto:Data: 2024-03-04T16:27:42.376 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.376 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.376 DEBUG:boto:Params: {} 2024-03-04T16:27:42.376 DEBUG:boto:Token: None 2024-03-04T16:27:42.376 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/?versions 2024-03-04T16:27:42.376 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc= 2024-03-04T16:27:42.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc=', 'Content-Length': '0'} 2024-03-04T16:27:42.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022edc77545cd2df1-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.420 DEBUG:boto:b'sbnvju-20obj-test-zone2999falseobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.421 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:27:42.421 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:27:42.421 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?acl 2024-03-04T16:27:42.421 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?acl 2024-03-04T16:27:42.421 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.421 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?acl 2024-03-04T16:27:42.421 DEBUG:boto:Data: 2024-03-04T16:27:42.421 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.422 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.422 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.422 DEBUG:boto:Params: {} 2024-03-04T16:27:42.422 DEBUG:boto:Token: None 2024-03-04T16:27:42.422 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-20/obj-test-zone2?acl 2024-03-04T16:27:42.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tFW5d9gg+sh7qOQVA/S9ydQXLdU= 2024-03-04T16:27:42.423 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tFW5d9gg+sh7qOQVA/S9ydQXLdU=', 'Content-Length': '0'} 2024-03-04T16:27:42.429 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd1b895f0144c5bf-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.430 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.430 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.430 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.430 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:27:42.430 DEBUG:boto:Data: 2024-03-04T16:27:42.430 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.430 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.430 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.431 DEBUG:boto:Params: {} 2024-03-04T16:27:42.431 DEBUG:boto:Token: None 2024-03-04T16:27:42.431 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/ 2024-03-04T16:27:42.431 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4= 2024-03-04T16:27:42.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0y+ZLTND9Be0YszWrhzfZTqP1e4=', 'Content-Length': '0'} 2024-03-04T16:27:42.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f347a3637b54f0f0-0065e5f67e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.434 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:27:42.434 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:27:42.434 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.434 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.434 DEBUG:boto:Method: GET 2024-03-04T16:27:42.434 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.434 DEBUG:boto:Data: 2024-03-04T16:27:42.434 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.435 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.435 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.435 DEBUG:boto:Params: {} 2024-03-04T16:27:42.435 DEBUG:boto:Token: None 2024-03-04T16:27:42.435 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-20/?versions 2024-03-04T16:27:42.435 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc= 2024-03-04T16:27:42.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gUwwAczy1HhtOQ1jHndd+FBbqJc=', 'Content-Length': '0'} 2024-03-04T16:27:42.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d52bbff701e44d8b-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.437 DEBUG:boto:b'sbnvju-20obj-test-zone3999falseobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.438 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:27:42.438 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:27:42.438 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?acl 2024-03-04T16:27:42.438 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?acl 2024-03-04T16:27:42.438 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.438 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?acl 2024-03-04T16:27:42.438 DEBUG:boto:Data: 2024-03-04T16:27:42.439 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.439 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.439 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.439 DEBUG:boto:Params: {} 2024-03-04T16:27:42.439 DEBUG:boto:Token: None 2024-03-04T16:27:42.439 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-20/obj-test-zone3?acl 2024-03-04T16:27:42.439 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KXOhY0SgDXI6aLvgl2EuGugpxKk= 2024-03-04T16:27:42.439 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KXOhY0SgDXI6aLvgl2EuGugpxKk=', 'Content-Length': '0'} 2024-03-04T16:27:42.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000348c3a83f27b2f5a-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.445 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.446 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.446 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.446 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:42.446 DEBUG:boto:Data: 2024-03-04T16:27:42.446 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.446 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.446 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.446 DEBUG:boto:Params: {} 2024-03-04T16:27:42.446 DEBUG:boto:Token: None 2024-03-04T16:27:42.446 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs= 2024-03-04T16:27:42.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs=', 'Content-Length': '0'} 2024-03-04T16:27:42.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000165b82b84a588b6a-0065e5f67e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.450 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.450 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.450 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.450 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.451 DEBUG:boto:Method: GET 2024-03-04T16:27:42.451 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-03-04T16:27:42.451 DEBUG:boto:Data: 2024-03-04T16:27:42.451 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.451 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.451 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.451 DEBUG:boto:Params: {} 2024-03-04T16:27:42.451 DEBUG:boto:Token: None 2024-03-04T16:27:42.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/?versions 2024-03-04T16:27:42.452 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug= 2024-03-04T16:27:42.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug=', 'Content-Length': '0'} 2024-03-04T16:27:42.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb7930ce15054937-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.496 DEBUG:boto:b'sbnvju-21obj-test-zone1999falseobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.496 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:42.497 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:27:42.497 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?acl 2024-03-04T16:27:42.497 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?acl 2024-03-04T16:27:42.497 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.497 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?acl 2024-03-04T16:27:42.497 DEBUG:boto:Data: 2024-03-04T16:27:42.497 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:27:42.497 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.497 DEBUG:boto:Params: {} 2024-03-04T16:27:42.498 DEBUG:boto:Token: None 2024-03-04T16:27:42.498 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-21/obj-test-zone1?acl 2024-03-04T16:27:42.498 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lJ1qeZxyrLHmUVt5Rbl7Bdjg6X0= 2024-03-04T16:27:42.498 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lJ1qeZxyrLHmUVt5Rbl7Bdjg6X0=', 'Content-Length': '0'} 2024-03-04T16:27:42.504 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5b4e524578f55bf-0065e5f67e-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.505 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.505 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.505 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.505 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:42.505 DEBUG:boto:Data: 2024-03-04T16:27:42.505 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.505 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.505 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.505 DEBUG:boto:Params: {} 2024-03-04T16:27:42.505 DEBUG:boto:Token: None 2024-03-04T16:27:42.506 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs= 2024-03-04T16:27:42.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs=', 'Content-Length': '0'} 2024-03-04T16:27:42.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009426aae51ca2f8b3-0065e5f67e-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.508 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.509 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.509 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.509 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.509 DEBUG:boto:Method: GET 2024-03-04T16:27:42.509 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-03-04T16:27:42.509 DEBUG:boto:Data: 2024-03-04T16:27:42.509 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.509 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.509 DEBUG:boto:Params: {} 2024-03-04T16:27:42.510 DEBUG:boto:Token: None 2024-03-04T16:27:42.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/?versions 2024-03-04T16:27:42.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug= 2024-03-04T16:27:42.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug=', 'Content-Length': '0'} 2024-03-04T16:27:42.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed0b4e434e69f67-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.513 DEBUG:boto:b'sbnvju-21obj-test-zone2999falseobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.513 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:42.513 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:27:42.514 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?acl 2024-03-04T16:27:42.514 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?acl 2024-03-04T16:27:42.514 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.514 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?acl 2024-03-04T16:27:42.514 DEBUG:boto:Data: 2024-03-04T16:27:42.514 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:27:42.515 DEBUG:boto:Port: 8000 2024-03-04T16:27:42.515 DEBUG:boto:Params: {} 2024-03-04T16:27:42.515 DEBUG:boto:Token: None 2024-03-04T16:27:42.515 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-21/obj-test-zone2?acl 2024-03-04T16:27:42.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YuSeVPFTXQNPECxhr1TWSCWRg1s= 2024-03-04T16:27:42.516 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YuSeVPFTXQNPECxhr1TWSCWRg1s=', 'Content-Length': '0'} 2024-03-04T16:27:42.523 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d472d4362be0649-0065e5f67e-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.523 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.523 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.523 DEBUG:boto:Method: HEAD 2024-03-04T16:27:42.523 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:27:42.523 DEBUG:boto:Data: 2024-03-04T16:27:42.523 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.523 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.524 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.524 DEBUG:boto:Params: {} 2024-03-04T16:27:42.524 DEBUG:boto:Token: None 2024-03-04T16:27:42.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/ 2024-03-04T16:27:42.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs= 2024-03-04T16:27:42.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xdAPfzDzYOSwFHfhmG6nU7MPHIs=', 'Content-Length': '0'} 2024-03-04T16:27:42.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e159c8ea15c7cc9-0065e5f67e-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.527 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:27:42.527 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:27:42.527 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.527 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.527 DEBUG:boto:Method: GET 2024-03-04T16:27:42.527 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-03-04T16:27:42.527 DEBUG:boto:Data: 2024-03-04T16:27:42.528 DEBUG:boto:Headers: {} 2024-03-04T16:27:42.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.528 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.528 DEBUG:boto:Params: {} 2024-03-04T16:27:42.528 DEBUG:boto:Token: None 2024-03-04T16:27:42.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:27:42 GMT /sbnvju-21/?versions 2024-03-04T16:27:42.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug= 2024-03-04T16:27:42.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YBVzKlO0DjUu2cBSsodaLOqYEug=', 'Content-Length': '0'} 2024-03-04T16:27:42.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1a2bc281f6e7ed-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.531 DEBUG:boto:b'sbnvju-21obj-test-zone3999falseobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:27:42.532 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.532 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:27:42.532 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?acl 2024-03-04T16:27:42.532 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?acl 2024-03-04T16:27:42.532 DEBUG:boto:Method: PUT 2024-03-04T16:27:42.532 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?acl 2024-03-04T16:27:42.532 DEBUG:boto:Data: 2024-03-04T16:27:42.532 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:27:42.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:27:42.533 DEBUG:boto:Port: 8001 2024-03-04T16:27:42.533 DEBUG:boto:Params: {} 2024-03-04T16:27:42.533 DEBUG:boto:Token: None 2024-03-04T16:27:42.533 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:27:42 GMT x-amz-acl:public-read /sbnvju-21/obj-test-zone3?acl 2024-03-04T16:27:42.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Yf+YRP3LG/dXJ4XsoL/cNdaCTzI= 2024-03-04T16:27:42.533 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:27:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Yf+YRP3LG/dXJ4XsoL/cNdaCTzI=', 'Content-Length': '0'} 2024-03-04T16:27:42.540 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005822d5a14d64c62f-0065e5f67e-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:27:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:27:42.540 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-19 2024-03-04T16:27:42.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:42.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:42.541 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:42.601 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:42.601 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:42.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:42.656+0000 7f365e416800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:42.656 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.43.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:42.656 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:47.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:47.657+0000 7f365e416800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:47.658 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.43.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:47.658 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:52.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:52.658+0000 7f365e416800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:52.659 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000010.43.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:52.659 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:57.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:57.659+0000 7f365e416800 1 bucket sync caught up with source: 2024-03-04T16:27:57.660 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:57.660 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:57.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:27:57.659+0000 7f365e416800 0 bucket checkpoint complete 2024-03-04T16:27:57.680 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-19 2024-03-04T16:27:57.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.681 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:57.741 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:57.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:57.796+0000 7f7472024800 1 bucket sync caught up with source: 2024-03-04T16:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000011.46.6, , 00000000003.20.10, , 00000000008.59.10, , , , , ] 2024-03-04T16:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:57.796+0000 7f7472024800 0 bucket checkpoint complete 2024-03-04T16:27:57.815 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-19 2024-03-04T16:27:57.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.815 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:57.871 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:57.871 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:57.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:57.922+0000 7fa81ca0f800 1 bucket sync caught up with source: 2024-03-04T16:27:57.922 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000011.39.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:27:57.922 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000011.39.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:27:57.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:57.922+0000 7fa81ca0f800 0 bucket checkpoint complete 2024-03-04T16:27:57.940 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-19 2024-03-04T16:27:57.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:27:57.941 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:27:57.995 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:27:57.995 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:27:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:27:58.045+0000 7f0dae20f800 1 waiting for incremental sync to catch up: 2024-03-04T16:27:58.045 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000010.36.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:27:58.046 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000011.39.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:28:03.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:03.046+0000 7f0dae20f800 1 bucket sync caught up with source: 2024-03-04T16:28:03.048 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000011.39.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:28:03.048 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000011.39.6, , 00000000006.48.10, , , , , ] 2024-03-04T16:28:03.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:03.046+0000 7f0dae20f800 0 bucket checkpoint complete 2024-03-04T16:28:03.072 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-19 2024-03-04T16:28:03.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:03.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:03.073 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:03.187+0000 7fa6cd211800 1 bucket sync caught up with source: 2024-03-04T16:28:03.188 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.23.10, , 00000000003.17.10, , 00000000015.67.6, , , , , ] 2024-03-04T16:28:03.188 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000015.67.6, , , , , ] 2024-03-04T16:28:03.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:03.187+0000 7fa6cd211800 0 bucket checkpoint complete 2024-03-04T16:28:03.208 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-19 2024-03-04T16:28:03.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:03.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:03.208 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:03.321+0000 7f3089c16800 1 bucket sync caught up with source: 2024-03-04T16:28:03.321 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.23.10, , 00000000003.17.10, , 00000000015.67.6, , , , , ] 2024-03-04T16:28:03.321 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.23.10, , 00000000003.17.10, , 00000000015.67.6, , , , , ] 2024-03-04T16:28:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:03.321+0000 7f3089c16800 0 bucket checkpoint complete 2024-03-04T16:28:03.340 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone2, test-zone1} 2024-03-04T16:28:03.340 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.340 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.340 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.340 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:03.341 DEBUG:boto:Data: 2024-03-04T16:28:03.341 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.341 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.341 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.341 DEBUG:boto:Params: {} 2024-03-04T16:28:03.341 DEBUG:boto:Token: None 2024-03-04T16:28:03.341 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.341 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RjyiSzrAwEk2RMgH3F50gPAgI78= 2024-03-04T16:28:03.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RjyiSzrAwEk2RMgH3F50gPAgI78=', 'Content-Length': '0'} 2024-03-04T16:28:03.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9832c05df752aec-0065e5f693-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.344 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.344 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.344 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.344 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:03.344 DEBUG:boto:Data: 2024-03-04T16:28:03.344 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.345 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.345 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.345 DEBUG:boto:Params: {} 2024-03-04T16:28:03.345 DEBUG:boto:Token: None 2024-03-04T16:28:03.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RjyiSzrAwEk2RMgH3F50gPAgI78= 2024-03-04T16:28:03.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RjyiSzrAwEk2RMgH3F50gPAgI78=', 'Content-Length': '0'} 2024-03-04T16:28:03.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000832fc49c5d6cfb32-0065e5f693-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:03.348 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.348 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.348 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.348 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.348 DEBUG:boto:Method: GET 2024-03-04T16:28:03.349 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.349 DEBUG:boto:Data: 2024-03-04T16:28:03.349 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.349 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.349 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.349 DEBUG:boto:Params: {} 2024-03-04T16:28:03.349 DEBUG:boto:Token: None 2024-03-04T16:28:03.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/?versions 2024-03-04T16:28:03.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo= 2024-03-04T16:28:03.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo=', 'Content-Length': '0'} 2024-03-04T16:28:03.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbdca734775ba4af-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.354 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.354 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:03.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:03.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:03.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:03.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:03.355 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.355 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.355 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.355 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.355 DEBUG:boto:Method: GET 2024-03-04T16:28:03.355 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.356 DEBUG:boto:Data: 2024-03-04T16:28:03.356 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.356 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.356 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.356 DEBUG:boto:Params: {} 2024-03-04T16:28:03.356 DEBUG:boto:Token: None 2024-03-04T16:28:03.356 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/?versions 2024-03-04T16:28:03.356 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo= 2024-03-04T16:28:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo=', 'Content-Length': '0'} 2024-03-04T16:28:03.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbb57f86a2be91e-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.359 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:03.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:03.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:03.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:03.360 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.360 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.360 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.360 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.360 DEBUG:boto:Method: GET 2024-03-04T16:28:03.360 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.360 DEBUG:boto:Data: 2024-03-04T16:28:03.360 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.360 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.360 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.361 DEBUG:boto:Params: {} 2024-03-04T16:28:03.361 DEBUG:boto:Token: None 2024-03-04T16:28:03.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/?versions 2024-03-04T16:28:03.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo= 2024-03-04T16:28:03.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo=', 'Content-Length': '0'} 2024-03-04T16:28:03.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f78d4788da8ceea-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.363 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.364 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.364 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.364 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.364 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.364 DEBUG:boto:Method: GET 2024-03-04T16:28:03.364 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:03.365 DEBUG:boto:Data: 2024-03-04T16:28:03.365 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.365 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.365 DEBUG:boto:Params: {} 2024-03-04T16:28:03.365 DEBUG:boto:Token: None 2024-03-04T16:28:03.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/?versions 2024-03-04T16:28:03.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo= 2024-03-04T16:28:03.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4BQ4qoBbsDJhzbCQ9KVbCyk+lGo=', 'Content-Length': '0'} 2024-03-04T16:28:03.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d0e2f5f2873e33-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.368 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:03.368 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.368 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.368 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.368 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.369 DEBUG:boto:Method: GET 2024-03-04T16:28:03.369 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.369 DEBUG:boto:Data: 2024-03-04T16:28:03.369 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.369 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.369 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.369 DEBUG:boto:Params: {} 2024-03-04T16:28:03.369 DEBUG:boto:Token: None 2024-03-04T16:28:03.369 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.369 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8cde9d4ca3167ea-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.371 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.372 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.372 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.372 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.372 DEBUG:boto:Method: GET 2024-03-04T16:28:03.372 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.372 DEBUG:boto:Data: 2024-03-04T16:28:03.372 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.373 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.373 DEBUG:boto:Params: {} 2024-03-04T16:28:03.373 DEBUG:boto:Token: None 2024-03-04T16:28:03.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001de92183f18a9c7c-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.375 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.375 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.375 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.375 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.375 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.375 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.376 DEBUG:boto:Data: 2024-03-04T16:28:03.376 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.376 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.376 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.376 DEBUG:boto:Params: {} 2024-03-04T16:28:03.376 DEBUG:boto:Token: None 2024-03-04T16:28:03.376 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.376 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5DzV1HMWxyBzZqqM8m18hsbIo/g= 2024-03-04T16:28:03.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5DzV1HMWxyBzZqqM8m18hsbIo/g=', 'Content-Length': '0'} 2024-03-04T16:28:03.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000946972af5c2d5901-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.379 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.379 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.379 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.379 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.379 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.379 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.379 DEBUG:boto:Data: 2024-03-04T16:28:03.379 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.380 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.380 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.380 DEBUG:boto:Params: {} 2024-03-04T16:28:03.380 DEBUG:boto:Token: None 2024-03-04T16:28:03.380 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.380 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5DzV1HMWxyBzZqqM8m18hsbIo/g= 2024-03-04T16:28:03.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5DzV1HMWxyBzZqqM8m18hsbIo/g=', 'Content-Length': '0'} 2024-03-04T16:28:03.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084975410c3c511d0-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:03.382 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.382 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.382 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.382 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.383 DEBUG:boto:Method: GET 2024-03-04T16:28:03.383 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.383 DEBUG:boto:Data: 2024-03-04T16:28:03.383 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.383 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.383 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.383 DEBUG:boto:Params: {} 2024-03-04T16:28:03.383 DEBUG:boto:Token: None 2024-03-04T16:28:03.383 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.383 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067272327d84f11f7-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.385 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.386 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.386 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.386 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.386 DEBUG:boto:Method: GET 2024-03-04T16:28:03.386 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.386 DEBUG:boto:Data: 2024-03-04T16:28:03.386 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.386 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.386 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.386 DEBUG:boto:Params: {} 2024-03-04T16:28:03.386 DEBUG:boto:Token: None 2024-03-04T16:28:03.386 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.387 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3dd1a672270230c-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.389 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.389 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.389 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.389 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.389 DEBUG:boto:Method: GET 2024-03-04T16:28:03.389 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.389 DEBUG:boto:Data: 2024-03-04T16:28:03.389 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.389 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.389 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.390 DEBUG:boto:Params: {} 2024-03-04T16:28:03.390 DEBUG:boto:Token: None 2024-03-04T16:28:03.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb94d33801300ae-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.392 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.392 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.393 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.393 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.393 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.393 DEBUG:boto:Method: GET 2024-03-04T16:28:03.393 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.393 DEBUG:boto:Data: 2024-03-04T16:28:03.393 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.393 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.393 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.393 DEBUG:boto:Params: {} 2024-03-04T16:28:03.393 DEBUG:boto:Token: None 2024-03-04T16:28:03.394 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188a551f618c146d-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.396 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.396 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.396 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.396 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.396 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:03.396 DEBUG:boto:Data: 2024-03-04T16:28:03.397 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.397 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.397 DEBUG:boto:Params: {} 2024-03-04T16:28:03.397 DEBUG:boto:Token: None 2024-03-04T16:28:03.397 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy 2024-03-04T16:28:03.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ihht4vJ9G1D3fPPNdLbKfN9Kg04= 2024-03-04T16:28:03.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ihht4vJ9G1D3fPPNdLbKfN9Kg04=', 'Content-Length': '0'} 2024-03-04T16:28:03.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002333475fea592781-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.399 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.399 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.399 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.400 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:03.400 DEBUG:boto:Data: 2024-03-04T16:28:03.400 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.400 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.400 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.400 DEBUG:boto:Params: {} 2024-03-04T16:28:03.400 DEBUG:boto:Token: None 2024-03-04T16:28:03.400 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy 2024-03-04T16:28:03.400 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ihht4vJ9G1D3fPPNdLbKfN9Kg04= 2024-03-04T16:28:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ihht4vJ9G1D3fPPNdLbKfN9Kg04=', 'Content-Length': '0'} 2024-03-04T16:28:03.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ab8fb3542060f8f-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:03.403 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.403 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.403 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.403 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.403 DEBUG:boto:Method: GET 2024-03-04T16:28:03.403 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.403 DEBUG:boto:Data: 2024-03-04T16:28:03.403 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.403 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.403 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.404 DEBUG:boto:Params: {} 2024-03-04T16:28:03.404 DEBUG:boto:Token: None 2024-03-04T16:28:03.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.404 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2b620cc7b7178d3-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.406 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:03.406 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:03.406 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.406 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.406 DEBUG:boto:Method: GET 2024-03-04T16:28:03.406 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.406 DEBUG:boto:Data: 2024-03-04T16:28:03.406 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.407 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.407 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.407 DEBUG:boto:Params: {} 2024-03-04T16:28:03.407 DEBUG:boto:Token: None 2024-03-04T16:28:03.407 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:03.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI= 2024-03-04T16:28:03.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8kkjovLJP49BNicgUE3TMkV8siI=', 'Content-Length': '0'} 2024-03-04T16:28:03.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6a3c8d41b205867-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.409 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.409 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.409 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.409 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.410 DEBUG:boto:Method: GET 2024-03-04T16:28:03.410 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.410 DEBUG:boto:Data: 2024-03-04T16:28:03.410 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.410 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.410 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.410 DEBUG:boto:Params: {} 2024-03-04T16:28:03.410 DEBUG:boto:Token: None 2024-03-04T16:28:03.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013632e675b6ab5f4-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.413 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.413 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.413 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.413 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.413 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.413 DEBUG:boto:Method: GET 2024-03-04T16:28:03.413 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:03.414 DEBUG:boto:Data: 2024-03-04T16:28:03.414 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.414 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.414 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.414 DEBUG:boto:Params: {} 2024-03-04T16:28:03.414 DEBUG:boto:Token: None 2024-03-04T16:28:03.415 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3254dd99fa23670-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.460 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:03.461 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.461 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.461 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.461 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.461 DEBUG:boto:Method: GET 2024-03-04T16:28:03.461 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.461 DEBUG:boto:Data: 2024-03-04T16:28:03.461 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.462 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.462 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.462 DEBUG:boto:Params: {} 2024-03-04T16:28:03.462 DEBUG:boto:Token: None 2024-03-04T16:28:03.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000727511f4fe094d77-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.464 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.465 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.465 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.465 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.465 DEBUG:boto:Method: GET 2024-03-04T16:28:03.465 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.465 DEBUG:boto:Data: 2024-03-04T16:28:03.466 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.466 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.466 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.466 DEBUG:boto:Params: {} 2024-03-04T16:28:03.466 DEBUG:boto:Token: None 2024-03-04T16:28:03.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa79193ddeefd8fa-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.468 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.468 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.469 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.469 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.469 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.469 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.469 DEBUG:boto:Data: 2024-03-04T16:28:03.469 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.469 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.470 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.470 DEBUG:boto:Params: {} 2024-03-04T16:28:03.470 DEBUG:boto:Token: None 2024-03-04T16:28:03.470 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IrzXFfYPwvvg1cPAy8RcPRnzsvI= 2024-03-04T16:28:03.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IrzXFfYPwvvg1cPAy8RcPRnzsvI=', 'Content-Length': '0'} 2024-03-04T16:28:03.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000144ba9b795b84c02-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.472 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.472 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.472 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.472 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.473 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.473 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.473 DEBUG:boto:Data: 2024-03-04T16:28:03.473 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.473 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.473 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.473 DEBUG:boto:Params: {} 2024-03-04T16:28:03.473 DEBUG:boto:Token: None 2024-03-04T16:28:03.473 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IrzXFfYPwvvg1cPAy8RcPRnzsvI= 2024-03-04T16:28:03.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IrzXFfYPwvvg1cPAy8RcPRnzsvI=', 'Content-Length': '0'} 2024-03-04T16:28:03.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4e701f7a0cb9280-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:03.475 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.476 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.476 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.476 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.476 DEBUG:boto:Method: GET 2024-03-04T16:28:03.476 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.476 DEBUG:boto:Data: 2024-03-04T16:28:03.476 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.476 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.476 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.476 DEBUG:boto:Params: {} 2024-03-04T16:28:03.477 DEBUG:boto:Token: None 2024-03-04T16:28:03.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ebcba2755d15d47-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.524 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.525 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.525 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.525 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.525 DEBUG:boto:Method: GET 2024-03-04T16:28:03.525 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.525 DEBUG:boto:Data: 2024-03-04T16:28:03.525 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.525 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.525 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.525 DEBUG:boto:Params: {} 2024-03-04T16:28:03.525 DEBUG:boto:Token: None 2024-03-04T16:28:03.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c38f17dcc0bb435-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.529 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.529 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.529 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.529 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.529 DEBUG:boto:Method: GET 2024-03-04T16:28:03.530 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.530 DEBUG:boto:Data: 2024-03-04T16:28:03.530 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.530 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.530 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.530 DEBUG:boto:Params: {} 2024-03-04T16:28:03.530 DEBUG:boto:Token: None 2024-03-04T16:28:03.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50f885be9479791-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.533 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.534 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.534 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.534 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.534 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.534 DEBUG:boto:Method: GET 2024-03-04T16:28:03.534 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.534 DEBUG:boto:Data: 2024-03-04T16:28:03.534 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.534 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.534 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.535 DEBUG:boto:Params: {} 2024-03-04T16:28:03.535 DEBUG:boto:Token: None 2024-03-04T16:28:03.536 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.536 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0b4b5d80d772ad-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.584 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.585 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.585 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.585 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.585 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.585 DEBUG:boto:Data: 2024-03-04T16:28:03.585 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.585 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.585 DEBUG:boto:Params: {} 2024-03-04T16:28:03.585 DEBUG:boto:Token: None 2024-03-04T16:28:03.586 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cBPpDl3rL2pwV/5ZfekCrWqos= 2024-03-04T16:28:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cBPpDl3rL2pwV/5ZfekCrWqos=', 'Content-Length': '0'} 2024-03-04T16:28:03.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6453e7cdf017368-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.588 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.589 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.589 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.589 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.589 DEBUG:boto:Data: 2024-03-04T16:28:03.589 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.589 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.589 DEBUG:boto:Params: {} 2024-03-04T16:28:03.589 DEBUG:boto:Token: None 2024-03-04T16:28:03.589 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4cBPpDl3rL2pwV/5ZfekCrWqos= 2024-03-04T16:28:03.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cBPpDl3rL2pwV/5ZfekCrWqos=', 'Content-Length': '0'} 2024-03-04T16:28:03.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa9beda13bb02861-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:03.592 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.592 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.592 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.592 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.592 DEBUG:boto:Method: GET 2024-03-04T16:28:03.592 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.593 DEBUG:boto:Data: 2024-03-04T16:28:03.593 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.593 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.593 DEBUG:boto:Params: {} 2024-03-04T16:28:03.593 DEBUG:boto:Token: None 2024-03-04T16:28:03.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.593 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e7e0a2d2fa54f1d-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.640 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.640 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:03.640 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.640 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.641 DEBUG:boto:Method: GET 2024-03-04T16:28:03.641 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.641 DEBUG:boto:Data: 2024-03-04T16:28:03.641 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.641 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.641 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.641 DEBUG:boto:Params: {} 2024-03-04T16:28:03.641 DEBUG:boto:Token: None 2024-03-04T16:28:03.641 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:03.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg= 2024-03-04T16:28:03.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZRsOeItIJh0gIlJAtXNwFFq2Rkg=', 'Content-Length': '0'} 2024-03-04T16:28:03.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9ce73abdae13f7f-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.644 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.644 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.645 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.645 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.645 DEBUG:boto:Method: GET 2024-03-04T16:28:03.645 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.645 DEBUG:boto:Data: 2024-03-04T16:28:03.645 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.645 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.645 DEBUG:boto:Params: {} 2024-03-04T16:28:03.646 DEBUG:boto:Token: None 2024-03-04T16:28:03.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c44e1c75b727b44f-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.649 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.650 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.650 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.650 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.650 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.650 DEBUG:boto:Method: GET 2024-03-04T16:28:03.650 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:03.651 DEBUG:boto:Data: 2024-03-04T16:28:03.651 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.651 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.651 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.651 DEBUG:boto:Params: {} 2024-03-04T16:28:03.651 DEBUG:boto:Token: None 2024-03-04T16:28:03.652 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.652 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000381cde64fe26b50c-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.656 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:03.657 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.657 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.657 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.657 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.657 DEBUG:boto:Method: GET 2024-03-04T16:28:03.657 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.657 DEBUG:boto:Data: 2024-03-04T16:28:03.657 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.657 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.657 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.657 DEBUG:boto:Params: {} 2024-03-04T16:28:03.658 DEBUG:boto:Token: None 2024-03-04T16:28:03.658 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.658 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000972ded69b3c64541-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.704 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.704 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.704 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.705 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.705 DEBUG:boto:Method: GET 2024-03-04T16:28:03.705 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.705 DEBUG:boto:Data: 2024-03-04T16:28:03.705 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.705 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.705 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.705 DEBUG:boto:Params: {} 2024-03-04T16:28:03.705 DEBUG:boto:Token: None 2024-03-04T16:28:03.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e884aa00995b2648-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.708 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.708 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.708 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.708 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.708 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.708 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.708 DEBUG:boto:Data: 2024-03-04T16:28:03.708 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.708 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.708 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.708 DEBUG:boto:Params: {} 2024-03-04T16:28:03.709 DEBUG:boto:Token: None 2024-03-04T16:28:03.709 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.709 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o7XOhzpVal/txwWSZDyrK03Nys8= 2024-03-04T16:28:03.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o7XOhzpVal/txwWSZDyrK03Nys8=', 'Content-Length': '0'} 2024-03-04T16:28:03.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ab480b7d60d234b-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.711 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.711 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.711 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.711 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.711 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.711 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.711 DEBUG:boto:Data: 2024-03-04T16:28:03.711 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.712 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.712 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.712 DEBUG:boto:Params: {} 2024-03-04T16:28:03.712 DEBUG:boto:Token: None 2024-03-04T16:28:03.712 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.712 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:o7XOhzpVal/txwWSZDyrK03Nys8= 2024-03-04T16:28:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o7XOhzpVal/txwWSZDyrK03Nys8=', 'Content-Length': '0'} 2024-03-04T16:28:03.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001efeebf2686dad0a-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:03.714 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.714 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.714 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.714 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.714 DEBUG:boto:Method: GET 2024-03-04T16:28:03.715 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.715 DEBUG:boto:Data: 2024-03-04T16:28:03.715 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.715 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.715 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.715 DEBUG:boto:Params: {} 2024-03-04T16:28:03.715 DEBUG:boto:Token: None 2024-03-04T16:28:03.715 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.715 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018325888c1dc0d12-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.721 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.721 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.721 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.721 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.721 DEBUG:boto:Method: GET 2024-03-04T16:28:03.721 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.721 DEBUG:boto:Data: 2024-03-04T16:28:03.721 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.721 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.722 DEBUG:boto:Params: {} 2024-03-04T16:28:03.722 DEBUG:boto:Token: None 2024-03-04T16:28:03.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008220e6eaf3af104d-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.764 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.764 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.764 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.764 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.765 DEBUG:boto:Method: GET 2024-03-04T16:28:03.765 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.765 DEBUG:boto:Data: 2024-03-04T16:28:03.765 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.765 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.765 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.765 DEBUG:boto:Params: {} 2024-03-04T16:28:03.765 DEBUG:boto:Token: None 2024-03-04T16:28:03.765 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc53373785758e6-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.768 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.769 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.769 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.769 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.769 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.769 DEBUG:boto:Method: GET 2024-03-04T16:28:03.769 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.769 DEBUG:boto:Data: 2024-03-04T16:28:03.769 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.769 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.769 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.770 DEBUG:boto:Params: {} 2024-03-04T16:28:03.770 DEBUG:boto:Token: None 2024-03-04T16:28:03.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.770 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070dbf93fb02346dc-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.820 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.821 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.821 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.821 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.821 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.821 DEBUG:boto:Data: 2024-03-04T16:28:03.821 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.821 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.821 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.822 DEBUG:boto:Params: {} 2024-03-04T16:28:03.822 DEBUG:boto:Token: None 2024-03-04T16:28:03.822 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.822 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bw50qto2TWq62jZbO1NAsILgoG4= 2024-03-04T16:28:03.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bw50qto2TWq62jZbO1NAsILgoG4=', 'Content-Length': '0'} 2024-03-04T16:28:03.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f026f7b55c09436f-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.825 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.825 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.825 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.825 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.825 DEBUG:boto:Data: 2024-03-04T16:28:03.825 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.825 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.826 DEBUG:boto:Params: {} 2024-03-04T16:28:03.826 DEBUG:boto:Token: None 2024-03-04T16:28:03.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.826 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bw50qto2TWq62jZbO1NAsILgoG4= 2024-03-04T16:28:03.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bw50qto2TWq62jZbO1NAsILgoG4=', 'Content-Length': '0'} 2024-03-04T16:28:03.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000578f9e4dcd870cf5-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:03.828 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.829 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.829 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.829 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.829 DEBUG:boto:Method: GET 2024-03-04T16:28:03.829 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.829 DEBUG:boto:Data: 2024-03-04T16:28:03.829 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.829 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.829 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.829 DEBUG:boto:Params: {} 2024-03-04T16:28:03.830 DEBUG:boto:Token: None 2024-03-04T16:28:03.830 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8e24288f976c58e-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.832 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.832 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:03.832 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.832 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.832 DEBUG:boto:Method: GET 2024-03-04T16:28:03.832 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.832 DEBUG:boto:Data: 2024-03-04T16:28:03.832 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.832 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.833 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.833 DEBUG:boto:Params: {} 2024-03-04T16:28:03.833 DEBUG:boto:Token: None 2024-03-04T16:28:03.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:03.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY= 2024-03-04T16:28:03.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DIeVSThRM7W7FLndZMa2nKOQ7lY=', 'Content-Length': '0'} 2024-03-04T16:28:03.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e38649797c8432e-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.876 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.876 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.877 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.877 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.877 DEBUG:boto:Method: GET 2024-03-04T16:28:03.877 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.877 DEBUG:boto:Data: 2024-03-04T16:28:03.877 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.877 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.877 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.877 DEBUG:boto:Params: {} 2024-03-04T16:28:03.877 DEBUG:boto:Token: None 2024-03-04T16:28:03.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb70f634dcd2e13-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.880 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.881 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:03.881 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:03.881 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.881 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.881 DEBUG:boto:Method: GET 2024-03-04T16:28:03.881 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:03.881 DEBUG:boto:Data: 2024-03-04T16:28:03.882 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.882 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.882 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.882 DEBUG:boto:Params: {} 2024-03-04T16:28:03.882 DEBUG:boto:Token: None 2024-03-04T16:28:03.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/ 2024-03-04T16:28:03.882 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg= 2024-03-04T16:28:03.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6QLxoYf9yb7IRw996lJ7WrfaYPg=', 'Content-Length': '0'} 2024-03-04T16:28:03.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae12388724342bc3-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.932 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:03.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:03.933 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.933 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.933 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.933 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.933 DEBUG:boto:Method: GET 2024-03-04T16:28:03.933 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.933 DEBUG:boto:Data: 2024-03-04T16:28:03.934 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.934 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.934 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.934 DEBUG:boto:Params: {} 2024-03-04T16:28:03.934 DEBUG:boto:Token: None 2024-03-04T16:28:03.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo= 2024-03-04T16:28:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo=', 'Content-Length': '0'} 2024-03-04T16:28:03.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002f20a20352591eb-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.980 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.980 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.981 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.981 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.981 DEBUG:boto:Method: GET 2024-03-04T16:28:03.981 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.981 DEBUG:boto:Data: 2024-03-04T16:28:03.981 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.981 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.981 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.981 DEBUG:boto:Params: {} 2024-03-04T16:28:03.981 DEBUG:boto:Token: None 2024-03-04T16:28:03.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.982 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo= 2024-03-04T16:28:03.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo=', 'Content-Length': '0'} 2024-03-04T16:28:03.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000998393fe34c66c13-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.984 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.984 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.984 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.984 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.984 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.985 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.985 DEBUG:boto:Data: 2024-03-04T16:28:03.985 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.985 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.985 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.985 DEBUG:boto:Params: {} 2024-03-04T16:28:03.985 DEBUG:boto:Token: None 2024-03-04T16:28:03.985 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.985 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KWPnJLL79FHPb05zH7QAbg6pLwE= 2024-03-04T16:28:03.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KWPnJLL79FHPb05zH7QAbg6pLwE=', 'Content-Length': '0'} 2024-03-04T16:28:03.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b539ef2788231eff-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.988 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.988 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.988 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.988 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.988 DEBUG:boto:Method: HEAD 2024-03-04T16:28:03.988 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.989 DEBUG:boto:Data: 2024-03-04T16:28:03.989 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:03.989 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.989 DEBUG:boto:Params: {} 2024-03-04T16:28:03.989 DEBUG:boto:Token: None 2024-03-04T16:28:03.989 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KWPnJLL79FHPb05zH7QAbg6pLwE= 2024-03-04T16:28:03.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KWPnJLL79FHPb05zH7QAbg6pLwE=', 'Content-Length': '0'} 2024-03-04T16:28:03.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d879e964f5d37b8-0065e5f693-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:03.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:03.993 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.993 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:03.993 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.993 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.993 DEBUG:boto:Method: GET 2024-03-04T16:28:03.993 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.993 DEBUG:boto:Data: 2024-03-04T16:28:03.993 DEBUG:boto:Headers: {} 2024-03-04T16:28:03.993 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:03.993 DEBUG:boto:Port: 8000 2024-03-04T16:28:03.993 DEBUG:boto:Params: {} 2024-03-04T16:28:03.994 DEBUG:boto:Token: None 2024-03-04T16:28:03.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:03 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:03.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo= 2024-03-04T16:28:03.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:03 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RnZFHhLmNS/sEl1v6fgqYFRb7Zo=', 'Content-Length': '0'} 2024-03-04T16:28:03.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca0d43a3b1759780-0065e5f693-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:03 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.036 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.036 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.036 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.037 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.037 DEBUG:boto:Method: GET 2024-03-04T16:28:04.037 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.037 DEBUG:boto:Data: 2024-03-04T16:28:04.037 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.037 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.037 DEBUG:boto:Params: {} 2024-03-04T16:28:04.037 DEBUG:boto:Token: None 2024-03-04T16:28:04.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o= 2024-03-04T16:28:04.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o=', 'Content-Length': '0'} 2024-03-04T16:28:04.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfc90dcbd23032e1-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.080 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.080 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.080 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.081 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.081 DEBUG:boto:Method: GET 2024-03-04T16:28:04.081 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.081 DEBUG:boto:Data: 2024-03-04T16:28:04.081 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.081 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:04.081 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.081 DEBUG:boto:Params: {} 2024-03-04T16:28:04.081 DEBUG:boto:Token: None 2024-03-04T16:28:04.082 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e49e6659242cdd5d-0065e5f694-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.085 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.085 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.085 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.085 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.085 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.085 DEBUG:boto:Method: GET 2024-03-04T16:28:04.086 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.086 DEBUG:boto:Data: 2024-03-04T16:28:04.086 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.086 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.086 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.086 DEBUG:boto:Params: {} 2024-03-04T16:28:04.086 DEBUG:boto:Token: None 2024-03-04T16:28:04.086 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.087 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060c9803b16e035d-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.089 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.090 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.090 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.090 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.090 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.090 DEBUG:boto:Data: 2024-03-04T16:28:04.090 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.090 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:04.090 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.090 DEBUG:boto:Params: {} 2024-03-04T16:28:04.091 DEBUG:boto:Token: None 2024-03-04T16:28:04.091 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.091 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSCkBcRJtrbHrBfD6qJ/8J5F3vQ= 2024-03-04T16:28:04.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSCkBcRJtrbHrBfD6qJ/8J5F3vQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf0aaffbd12837b4-0065e5f694-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.094 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.094 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.094 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.094 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.094 DEBUG:boto:Data: 2024-03-04T16:28:04.094 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.094 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.094 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.095 DEBUG:boto:Params: {} 2024-03-04T16:28:04.095 DEBUG:boto:Token: None 2024-03-04T16:28:04.095 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.095 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSCkBcRJtrbHrBfD6qJ/8J5F3vQ= 2024-03-04T16:28:04.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSCkBcRJtrbHrBfD6qJ/8J5F3vQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b56858d6d5a27a6a-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:04.100 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.100 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.101 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.101 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.101 DEBUG:boto:Method: GET 2024-03-04T16:28:04.101 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.101 DEBUG:boto:Data: 2024-03-04T16:28:04.101 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.101 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:04.101 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.102 DEBUG:boto:Params: {} 2024-03-04T16:28:04.102 DEBUG:boto:Token: None 2024-03-04T16:28:04.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o= 2024-03-04T16:28:04.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o=', 'Content-Length': '0'} 2024-03-04T16:28:04.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb99beea8726909d-0065e5f694-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.148 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.149 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:04.149 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.149 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.149 DEBUG:boto:Method: GET 2024-03-04T16:28:04.149 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.149 DEBUG:boto:Data: 2024-03-04T16:28:04.149 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.150 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.150 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.150 DEBUG:boto:Params: {} 2024-03-04T16:28:04.150 DEBUG:boto:Token: None 2024-03-04T16:28:04.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:04.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o= 2024-03-04T16:28:04.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pIRWh7ZGVjdVcuPzd48yPCIJ+2o=', 'Content-Length': '0'} 2024-03-04T16:28:04.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000596124528bac199f-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.196 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.196 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.197 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.197 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.197 DEBUG:boto:Method: GET 2024-03-04T16:28:04.197 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.197 DEBUG:boto:Data: 2024-03-04T16:28:04.197 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.197 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:04.197 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.197 DEBUG:boto:Params: {} 2024-03-04T16:28:04.197 DEBUG:boto:Token: None 2024-03-04T16:28:04.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006121c99cddf1089c-0065e5f694-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.244 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.245 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.245 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.245 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.245 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.245 DEBUG:boto:Method: GET 2024-03-04T16:28:04.245 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:04.245 DEBUG:boto:Data: 2024-03-04T16:28:04.246 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.246 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.246 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.246 DEBUG:boto:Params: {} 2024-03-04T16:28:04.246 DEBUG:boto:Token: None 2024-03-04T16:28:04.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003386f80775f040d1-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.249 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone2, test-zone1} 2024-03-04T16:28:04.250 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone3, test-zone1} 2024-03-04T16:28:04.250 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.250 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.251 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.251 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:04.251 DEBUG:boto:Data: 2024-03-04T16:28:04.251 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.251 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.251 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.251 DEBUG:boto:Params: {} 2024-03-04T16:28:04.251 DEBUG:boto:Token: None 2024-03-04T16:28:04.251 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.251 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KTJ9M/cs2yDjVQC4VTDLSLUHMT4= 2024-03-04T16:28:04.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KTJ9M/cs2yDjVQC4VTDLSLUHMT4=', 'Content-Length': '0'} 2024-03-04T16:28:04.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099ad6901a8dec9c1-0065e5f694-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.255 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.255 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.255 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.255 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:04.255 DEBUG:boto:Data: 2024-03-04T16:28:04.256 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.256 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.256 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.256 DEBUG:boto:Params: {} 2024-03-04T16:28:04.256 DEBUG:boto:Token: None 2024-03-04T16:28:04.256 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KTJ9M/cs2yDjVQC4VTDLSLUHMT4= 2024-03-04T16:28:04.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KTJ9M/cs2yDjVQC4VTDLSLUHMT4=', 'Content-Length': '0'} 2024-03-04T16:28:04.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec0d89fe298aa1af-0065e5f694-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.260 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:04.260 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.260 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.260 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.260 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.260 DEBUG:boto:Method: GET 2024-03-04T16:28:04.261 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.261 DEBUG:boto:Data: 2024-03-04T16:28:04.261 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.261 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.261 DEBUG:boto:Params: {} 2024-03-04T16:28:04.261 DEBUG:boto:Token: None 2024-03-04T16:28:04.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/?versions 2024-03-04T16:28:04.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ= 2024-03-04T16:28:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000533923ddcbe05422-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.264 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:04.265 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:04.265 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.265 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.265 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.265 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.266 DEBUG:boto:Method: GET 2024-03-04T16:28:04.266 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.266 DEBUG:boto:Data: 2024-03-04T16:28:04.266 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.266 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.266 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.266 DEBUG:boto:Params: {} 2024-03-04T16:28:04.266 DEBUG:boto:Token: None 2024-03-04T16:28:04.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/?versions 2024-03-04T16:28:04.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ= 2024-03-04T16:28:04.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717e953494f2bb55-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.269 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:04.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:04.270 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.271 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.271 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.271 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.271 DEBUG:boto:Method: GET 2024-03-04T16:28:04.271 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.271 DEBUG:boto:Data: 2024-03-04T16:28:04.271 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.271 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.271 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.271 DEBUG:boto:Params: {} 2024-03-04T16:28:04.271 DEBUG:boto:Token: None 2024-03-04T16:28:04.272 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/?versions 2024-03-04T16:28:04.272 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ= 2024-03-04T16:28:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5b563f698997473-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.275 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.276 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.276 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.276 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.276 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.276 DEBUG:boto:Method: GET 2024-03-04T16:28:04.276 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:04.276 DEBUG:boto:Data: 2024-03-04T16:28:04.276 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.276 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.276 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.277 DEBUG:boto:Params: {} 2024-03-04T16:28:04.277 DEBUG:boto:Token: None 2024-03-04T16:28:04.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/?versions 2024-03-04T16:28:04.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ= 2024-03-04T16:28:04.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iAx3A7c47oDDiCCtYb4tYcXr5jQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6cc9646880d532a-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.282 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:04.282 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.282 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.283 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.283 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.283 DEBUG:boto:Method: GET 2024-03-04T16:28:04.283 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.283 DEBUG:boto:Data: 2024-03-04T16:28:04.283 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.283 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.283 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.283 DEBUG:boto:Params: {} 2024-03-04T16:28:04.283 DEBUG:boto:Token: None 2024-03-04T16:28:04.283 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093dfd0a7467a6a24-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.286 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.286 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.286 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.287 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.287 DEBUG:boto:Method: GET 2024-03-04T16:28:04.287 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.287 DEBUG:boto:Data: 2024-03-04T16:28:04.287 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.287 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.287 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.287 DEBUG:boto:Params: {} 2024-03-04T16:28:04.287 DEBUG:boto:Token: None 2024-03-04T16:28:04.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030b4a2c10380840b-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.290 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.290 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.290 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.290 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.291 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.291 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.291 DEBUG:boto:Data: 2024-03-04T16:28:04.291 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.291 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.291 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.291 DEBUG:boto:Params: {} 2024-03-04T16:28:04.291 DEBUG:boto:Token: None 2024-03-04T16:28:04.292 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.292 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UhwzyxL3H7DROIs7vZlLCodYYJA= 2024-03-04T16:28:04.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UhwzyxL3H7DROIs7vZlLCodYYJA=', 'Content-Length': '0'} 2024-03-04T16:28:04.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000436897b701dfcba3-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.294 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.295 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.295 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.295 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.295 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.295 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.295 DEBUG:boto:Data: 2024-03-04T16:28:04.295 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.295 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.295 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.295 DEBUG:boto:Params: {} 2024-03-04T16:28:04.296 DEBUG:boto:Token: None 2024-03-04T16:28:04.296 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.296 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UhwzyxL3H7DROIs7vZlLCodYYJA= 2024-03-04T16:28:04.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UhwzyxL3H7DROIs7vZlLCodYYJA=', 'Content-Length': '0'} 2024-03-04T16:28:04.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4388efee98846b3-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:04.298 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.298 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.299 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.299 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.299 DEBUG:boto:Method: GET 2024-03-04T16:28:04.299 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.299 DEBUG:boto:Data: 2024-03-04T16:28:04.299 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.299 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.299 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.299 DEBUG:boto:Params: {} 2024-03-04T16:28:04.299 DEBUG:boto:Token: None 2024-03-04T16:28:04.299 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.300 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7ac30cdd4c2a133-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.301 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.302 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.302 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.302 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.302 DEBUG:boto:Method: GET 2024-03-04T16:28:04.302 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.303 DEBUG:boto:Data: 2024-03-04T16:28:04.303 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.303 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.303 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.303 DEBUG:boto:Params: {} 2024-03-04T16:28:04.303 DEBUG:boto:Token: None 2024-03-04T16:28:04.303 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.303 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000634f72a85ca4caab-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.307 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.307 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.307 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.307 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.307 DEBUG:boto:Method: GET 2024-03-04T16:28:04.307 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.308 DEBUG:boto:Data: 2024-03-04T16:28:04.308 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.308 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.308 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.308 DEBUG:boto:Params: {} 2024-03-04T16:28:04.308 DEBUG:boto:Token: None 2024-03-04T16:28:04.308 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.309 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e982af4cdf7ed4ea-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.352 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.353 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.353 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.353 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.353 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.353 DEBUG:boto:Method: GET 2024-03-04T16:28:04.353 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.353 DEBUG:boto:Data: 2024-03-04T16:28:04.353 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.353 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.354 DEBUG:boto:Params: {} 2024-03-04T16:28:04.354 DEBUG:boto:Token: None 2024-03-04T16:28:04.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009115ac17875625a7-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.357 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.357 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.358 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.358 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.358 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:04.358 DEBUG:boto:Data: 2024-03-04T16:28:04.358 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.358 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.358 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.358 DEBUG:boto:Params: {} 2024-03-04T16:28:04.358 DEBUG:boto:Token: None 2024-03-04T16:28:04.359 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy 2024-03-04T16:28:04.359 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eM8aTizQv65W9SIzhRYRi1IqtA4= 2024-03-04T16:28:04.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eM8aTizQv65W9SIzhRYRi1IqtA4=', 'Content-Length': '0'} 2024-03-04T16:28:04.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a801c01af5d21253-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.362 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.362 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.362 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.362 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:04.362 DEBUG:boto:Data: 2024-03-04T16:28:04.362 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.362 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.362 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.362 DEBUG:boto:Params: {} 2024-03-04T16:28:04.363 DEBUG:boto:Token: None 2024-03-04T16:28:04.363 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy 2024-03-04T16:28:04.363 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eM8aTizQv65W9SIzhRYRi1IqtA4= 2024-03-04T16:28:04.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eM8aTizQv65W9SIzhRYRi1IqtA4=', 'Content-Length': '0'} 2024-03-04T16:28:04.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acdd4429b1d9aeff-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:04.366 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.366 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.366 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.366 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.366 DEBUG:boto:Method: GET 2024-03-04T16:28:04.366 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.366 DEBUG:boto:Data: 2024-03-04T16:28:04.366 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.367 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.367 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.367 DEBUG:boto:Params: {} 2024-03-04T16:28:04.367 DEBUG:boto:Token: None 2024-03-04T16:28:04.367 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.367 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d712f2f9ddd3975d-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.369 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:04.369 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:04.369 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.369 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.370 DEBUG:boto:Method: GET 2024-03-04T16:28:04.370 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.370 DEBUG:boto:Data: 2024-03-04T16:28:04.370 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.370 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.370 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.370 DEBUG:boto:Params: {} 2024-03-04T16:28:04.370 DEBUG:boto:Token: None 2024-03-04T16:28:04.370 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:04.371 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg= 2024-03-04T16:28:04.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjEiMr6Ob1ndH+aeTQ0wX9/rgCg=', 'Content-Length': '0'} 2024-03-04T16:28:04.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008805c424c50bb71d-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.373 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.373 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.373 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.373 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.373 DEBUG:boto:Method: GET 2024-03-04T16:28:04.373 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.373 DEBUG:boto:Data: 2024-03-04T16:28:04.373 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.374 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.374 DEBUG:boto:Params: {} 2024-03-04T16:28:04.374 DEBUG:boto:Token: None 2024-03-04T16:28:04.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.374 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed3af996c0c7ee79-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.424 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.425 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.425 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.425 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.425 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.425 DEBUG:boto:Method: GET 2024-03-04T16:28:04.425 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:04.425 DEBUG:boto:Data: 2024-03-04T16:28:04.425 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.425 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.425 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.425 DEBUG:boto:Params: {} 2024-03-04T16:28:04.426 DEBUG:boto:Token: None 2024-03-04T16:28:04.426 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000247febed82b563e3-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.428 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:04.429 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.429 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.429 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.429 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.429 DEBUG:boto:Method: GET 2024-03-04T16:28:04.429 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.429 DEBUG:boto:Data: 2024-03-04T16:28:04.429 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.430 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.430 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.430 DEBUG:boto:Params: {} 2024-03-04T16:28:04.430 DEBUG:boto:Token: None 2024-03-04T16:28:04.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000361ed72aed05ee3d-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.480 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.480 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.481 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.481 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.481 DEBUG:boto:Method: GET 2024-03-04T16:28:04.481 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.481 DEBUG:boto:Data: 2024-03-04T16:28:04.481 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.481 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.481 DEBUG:boto:Params: {} 2024-03-04T16:28:04.481 DEBUG:boto:Token: None 2024-03-04T16:28:04.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e875e78e87969b45-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.484 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.484 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.484 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.484 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.484 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.484 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.484 DEBUG:boto:Data: 2024-03-04T16:28:04.485 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.485 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.485 DEBUG:boto:Params: {} 2024-03-04T16:28:04.485 DEBUG:boto:Token: None 2024-03-04T16:28:04.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.485 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sc6cjbRg1pok99bJq1Wk/sPQyp0= 2024-03-04T16:28:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sc6cjbRg1pok99bJq1Wk/sPQyp0=', 'Content-Length': '0'} 2024-03-04T16:28:04.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000708c585ed9cbe426-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.487 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.487 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.487 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.488 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.488 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.488 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.488 DEBUG:boto:Data: 2024-03-04T16:28:04.488 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.488 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.488 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.488 DEBUG:boto:Params: {} 2024-03-04T16:28:04.488 DEBUG:boto:Token: None 2024-03-04T16:28:04.488 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sc6cjbRg1pok99bJq1Wk/sPQyp0= 2024-03-04T16:28:04.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sc6cjbRg1pok99bJq1Wk/sPQyp0=', 'Content-Length': '0'} 2024-03-04T16:28:04.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f846db8406d3d0f5-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:04.492 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.492 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.492 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.492 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.492 DEBUG:boto:Method: GET 2024-03-04T16:28:04.492 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.493 DEBUG:boto:Data: 2024-03-04T16:28:04.493 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.493 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.493 DEBUG:boto:Params: {} 2024-03-04T16:28:04.493 DEBUG:boto:Token: None 2024-03-04T16:28:04.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da7146589bc15148-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.536 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.536 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.536 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.536 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.537 DEBUG:boto:Method: GET 2024-03-04T16:28:04.537 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.537 DEBUG:boto:Data: 2024-03-04T16:28:04.537 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.537 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.537 DEBUG:boto:Params: {} 2024-03-04T16:28:04.537 DEBUG:boto:Token: None 2024-03-04T16:28:04.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.537 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dfa9d54a8f4bc71-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.540 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.540 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.540 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.540 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.540 DEBUG:boto:Method: GET 2024-03-04T16:28:04.540 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.541 DEBUG:boto:Data: 2024-03-04T16:28:04.541 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.541 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.541 DEBUG:boto:Params: {} 2024-03-04T16:28:04.541 DEBUG:boto:Token: None 2024-03-04T16:28:04.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.541 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cff273952d4121a-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.584 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.585 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.585 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.585 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.585 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.585 DEBUG:boto:Method: GET 2024-03-04T16:28:04.585 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.585 DEBUG:boto:Data: 2024-03-04T16:28:04.585 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.585 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.585 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.585 DEBUG:boto:Params: {} 2024-03-04T16:28:04.586 DEBUG:boto:Token: None 2024-03-04T16:28:04.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002561c9ded891df74-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.589 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.589 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.589 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.589 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.589 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.589 DEBUG:boto:Data: 2024-03-04T16:28:04.589 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.590 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.590 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.590 DEBUG:boto:Params: {} 2024-03-04T16:28:04.590 DEBUG:boto:Token: None 2024-03-04T16:28:04.590 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1+F71T64mvwYlui6ylOHjH2K3OA= 2024-03-04T16:28:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1+F71T64mvwYlui6ylOHjH2K3OA=', 'Content-Length': '0'} 2024-03-04T16:28:04.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081935b7d5c6bb995-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.592 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.593 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.593 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.593 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.593 DEBUG:boto:Data: 2024-03-04T16:28:04.593 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.593 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.593 DEBUG:boto:Params: {} 2024-03-04T16:28:04.593 DEBUG:boto:Token: None 2024-03-04T16:28:04.593 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1+F71T64mvwYlui6ylOHjH2K3OA= 2024-03-04T16:28:04.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1+F71T64mvwYlui6ylOHjH2K3OA=', 'Content-Length': '0'} 2024-03-04T16:28:04.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aef4826a316cb149-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:04.596 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.597 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.597 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.597 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.597 DEBUG:boto:Method: GET 2024-03-04T16:28:04.597 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.597 DEBUG:boto:Data: 2024-03-04T16:28:04.597 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.598 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.598 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.598 DEBUG:boto:Params: {} 2024-03-04T16:28:04.598 DEBUG:boto:Token: None 2024-03-04T16:28:04.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014c3cee8b37b614d-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.644 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.644 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:04.644 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.645 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.645 DEBUG:boto:Method: GET 2024-03-04T16:28:04.645 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.645 DEBUG:boto:Data: 2024-03-04T16:28:04.645 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.645 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.645 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.645 DEBUG:boto:Params: {} 2024-03-04T16:28:04.645 DEBUG:boto:Token: None 2024-03-04T16:28:04.645 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:04.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw= 2024-03-04T16:28:04.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wF9XT5IW68wo2RpyLlBvr242ofw=', 'Content-Length': '0'} 2024-03-04T16:28:04.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000706ae2d530fa4c1f-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.648 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.648 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.648 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.648 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.648 DEBUG:boto:Method: GET 2024-03-04T16:28:04.648 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.648 DEBUG:boto:Data: 2024-03-04T16:28:04.649 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.649 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.649 DEBUG:boto:Params: {} 2024-03-04T16:28:04.649 DEBUG:boto:Token: None 2024-03-04T16:28:04.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad29d19e745d7220-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.692 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.693 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.693 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.693 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.693 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.693 DEBUG:boto:Method: GET 2024-03-04T16:28:04.693 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:04.693 DEBUG:boto:Data: 2024-03-04T16:28:04.694 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.694 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.694 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.694 DEBUG:boto:Params: {} 2024-03-04T16:28:04.694 DEBUG:boto:Token: None 2024-03-04T16:28:04.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b22d90aad25c8da-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.697 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:04.698 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.698 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.698 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.698 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.698 DEBUG:boto:Method: GET 2024-03-04T16:28:04.698 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.698 DEBUG:boto:Data: 2024-03-04T16:28:04.698 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.698 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.698 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.698 DEBUG:boto:Params: {} 2024-03-04T16:28:04.699 DEBUG:boto:Token: None 2024-03-04T16:28:04.699 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.699 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fffc0424f4fbfa67-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.703 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.703 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.703 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.703 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.703 DEBUG:boto:Method: GET 2024-03-04T16:28:04.703 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.703 DEBUG:boto:Data: 2024-03-04T16:28:04.704 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.704 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.704 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.704 DEBUG:boto:Params: {} 2024-03-04T16:28:04.704 DEBUG:boto:Token: None 2024-03-04T16:28:04.704 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.704 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b63082c9fcf5f475-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.707 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.707 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.707 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.707 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.708 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.708 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.708 DEBUG:boto:Data: 2024-03-04T16:28:04.708 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.708 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.709 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.709 DEBUG:boto:Params: {} 2024-03-04T16:28:04.709 DEBUG:boto:Token: None 2024-03-04T16:28:04.709 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.709 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u2XO6QG6f4WJgLq8+5XBHSwnwgs= 2024-03-04T16:28:04.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u2XO6QG6f4WJgLq8+5XBHSwnwgs=', 'Content-Length': '0'} 2024-03-04T16:28:04.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000436805bbb701e66f-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.712 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.712 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.712 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.712 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.712 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.712 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.712 DEBUG:boto:Data: 2024-03-04T16:28:04.712 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.712 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.712 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.713 DEBUG:boto:Params: {} 2024-03-04T16:28:04.713 DEBUG:boto:Token: None 2024-03-04T16:28:04.713 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.713 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u2XO6QG6f4WJgLq8+5XBHSwnwgs= 2024-03-04T16:28:04.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u2XO6QG6f4WJgLq8+5XBHSwnwgs=', 'Content-Length': '0'} 2024-03-04T16:28:04.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025ab4e580c9c575f-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:04.715 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.715 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.716 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.716 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.716 DEBUG:boto:Method: GET 2024-03-04T16:28:04.716 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.716 DEBUG:boto:Data: 2024-03-04T16:28:04.716 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.716 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.716 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.716 DEBUG:boto:Params: {} 2024-03-04T16:28:04.716 DEBUG:boto:Token: None 2024-03-04T16:28:04.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.717 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0ba335c3e1ea53d-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.760 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.760 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.760 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.760 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.760 DEBUG:boto:Method: GET 2024-03-04T16:28:04.761 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.761 DEBUG:boto:Data: 2024-03-04T16:28:04.761 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.761 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.761 DEBUG:boto:Params: {} 2024-03-04T16:28:04.761 DEBUG:boto:Token: None 2024-03-04T16:28:04.761 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.761 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efbc0ae505f21d3c-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.804 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.804 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.804 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.804 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.805 DEBUG:boto:Method: GET 2024-03-04T16:28:04.805 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.805 DEBUG:boto:Data: 2024-03-04T16:28:04.805 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.805 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.805 DEBUG:boto:Params: {} 2024-03-04T16:28:04.805 DEBUG:boto:Token: None 2024-03-04T16:28:04.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030b687cf7c4325be-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.853 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.854 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.854 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.854 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.854 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.854 DEBUG:boto:Method: GET 2024-03-04T16:28:04.854 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.854 DEBUG:boto:Data: 2024-03-04T16:28:04.854 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.855 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.855 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.855 DEBUG:boto:Params: {} 2024-03-04T16:28:04.855 DEBUG:boto:Token: None 2024-03-04T16:28:04.855 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7974f647f6130a6-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.858 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.859 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.859 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.859 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.859 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.859 DEBUG:boto:Data: 2024-03-04T16:28:04.859 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.859 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.860 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.860 DEBUG:boto:Params: {} 2024-03-04T16:28:04.860 DEBUG:boto:Token: None 2024-03-04T16:28:04.860 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Co9gt9H3KiRTgjlMkmlwKYZFSjg= 2024-03-04T16:28:04.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Co9gt9H3KiRTgjlMkmlwKYZFSjg=', 'Content-Length': '0'} 2024-03-04T16:28:04.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000280fc848280d6933-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.863 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.863 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.864 DEBUG:boto:Method: HEAD 2024-03-04T16:28:04.864 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.865 DEBUG:boto:Data: 2024-03-04T16:28:04.865 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.865 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.865 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.865 DEBUG:boto:Params: {} 2024-03-04T16:28:04.865 DEBUG:boto:Token: None 2024-03-04T16:28:04.865 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Co9gt9H3KiRTgjlMkmlwKYZFSjg= 2024-03-04T16:28:04.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Co9gt9H3KiRTgjlMkmlwKYZFSjg=', 'Content-Length': '0'} 2024-03-04T16:28:04.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f29e40a81cd8738f-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:04.869 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.869 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.869 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.869 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.869 DEBUG:boto:Method: GET 2024-03-04T16:28:04.869 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.869 DEBUG:boto:Data: 2024-03-04T16:28:04.869 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.869 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.869 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.870 DEBUG:boto:Params: {} 2024-03-04T16:28:04.870 DEBUG:boto:Token: None 2024-03-04T16:28:04.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.870 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090b91fc290c615b6-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.912 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.912 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:04.912 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.912 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.913 DEBUG:boto:Method: GET 2024-03-04T16:28:04.913 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.913 DEBUG:boto:Data: 2024-03-04T16:28:04.913 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.913 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.913 DEBUG:boto:Params: {} 2024-03-04T16:28:04.913 DEBUG:boto:Token: None 2024-03-04T16:28:04.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:04.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ= 2024-03-04T16:28:04.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QTLFU0FBoL+geXI3nMYz00pUlQ=', 'Content-Length': '0'} 2024-03-04T16:28:04.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000313b8416afcc67cc-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.960 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.960 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.960 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.960 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.961 DEBUG:boto:Method: GET 2024-03-04T16:28:04.961 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.961 DEBUG:boto:Data: 2024-03-04T16:28:04.961 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:04.961 DEBUG:boto:Port: 8001 2024-03-04T16:28:04.961 DEBUG:boto:Params: {} 2024-03-04T16:28:04.961 DEBUG:boto:Token: None 2024-03-04T16:28:04.961 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052ab1f0b765b3737-0065e5f694-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:04.965 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:04.965 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:04.965 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:04.966 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.966 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.966 DEBUG:boto:Method: GET 2024-03-04T16:28:04.966 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:04.966 DEBUG:boto:Data: 2024-03-04T16:28:04.966 DEBUG:boto:Headers: {} 2024-03-04T16:28:04.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:04.966 DEBUG:boto:Port: 8000 2024-03-04T16:28:04.966 DEBUG:boto:Params: {} 2024-03-04T16:28:04.966 DEBUG:boto:Token: None 2024-03-04T16:28:04.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:04 GMT /sbnvju-19/ 2024-03-04T16:28:04.967 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk= 2024-03-04T16:28:04.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/TYGkwCIVsoBUJ11nGXmIy4m3Bk=', 'Content-Length': '0'} 2024-03-04T16:28:04.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a45844c0acd7b57-0065e5f694-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.012 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:05.013 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.013 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.013 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.013 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.013 DEBUG:boto:Method: GET 2024-03-04T16:28:05.013 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.013 DEBUG:boto:Data: 2024-03-04T16:28:05.013 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.013 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.013 DEBUG:boto:Params: {} 2024-03-04T16:28:05.015 DEBUG:boto:Token: None 2024-03-04T16:28:05.015 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060388aa13f4a207a-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.060 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.061 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.061 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.061 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.061 DEBUG:boto:Method: GET 2024-03-04T16:28:05.061 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.061 DEBUG:boto:Data: 2024-03-04T16:28:05.061 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.061 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.061 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.062 DEBUG:boto:Params: {} 2024-03-04T16:28:05.062 DEBUG:boto:Token: None 2024-03-04T16:28:05.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b7057cacb685a9a-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.067 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.067 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.067 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.067 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.067 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.067 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.067 DEBUG:boto:Data: 2024-03-04T16:28:05.068 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.068 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.068 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.068 DEBUG:boto:Params: {} 2024-03-04T16:28:05.068 DEBUG:boto:Token: None 2024-03-04T16:28:05.069 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OfKamoLuKx/eUjFipt1clPZJMkI= 2024-03-04T16:28:05.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OfKamoLuKx/eUjFipt1clPZJMkI=', 'Content-Length': '0'} 2024-03-04T16:28:05.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd20a447df8323ff-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.072 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.072 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.072 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.072 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.072 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.072 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.072 DEBUG:boto:Data: 2024-03-04T16:28:05.073 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.073 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.073 DEBUG:boto:Params: {} 2024-03-04T16:28:05.073 DEBUG:boto:Token: None 2024-03-04T16:28:05.073 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.073 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OfKamoLuKx/eUjFipt1clPZJMkI= 2024-03-04T16:28:05.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OfKamoLuKx/eUjFipt1clPZJMkI=', 'Content-Length': '0'} 2024-03-04T16:28:05.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d75490248452b27-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:05.076 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.076 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.076 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.076 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.076 DEBUG:boto:Method: GET 2024-03-04T16:28:05.076 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.076 DEBUG:boto:Data: 2024-03-04T16:28:05.076 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.076 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.077 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.077 DEBUG:boto:Params: {} 2024-03-04T16:28:05.077 DEBUG:boto:Token: None 2024-03-04T16:28:05.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d5a1d5499105fec-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.079 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.080 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.080 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.080 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.080 DEBUG:boto:Method: GET 2024-03-04T16:28:05.080 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.080 DEBUG:boto:Data: 2024-03-04T16:28:05.080 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.080 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.080 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.080 DEBUG:boto:Params: {} 2024-03-04T16:28:05.081 DEBUG:boto:Token: None 2024-03-04T16:28:05.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027cbb94ec0b7684e-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.124 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.124 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.124 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.124 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.125 DEBUG:boto:Method: GET 2024-03-04T16:28:05.125 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.125 DEBUG:boto:Data: 2024-03-04T16:28:05.125 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.125 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.125 DEBUG:boto:Params: {} 2024-03-04T16:28:05.125 DEBUG:boto:Token: None 2024-03-04T16:28:05.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5145882a12f13bd-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.129 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.129 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.129 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.129 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.129 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.129 DEBUG:boto:Method: GET 2024-03-04T16:28:05.130 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.130 DEBUG:boto:Data: 2024-03-04T16:28:05.130 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.130 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.130 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.130 DEBUG:boto:Params: {} 2024-03-04T16:28:05.131 DEBUG:boto:Token: None 2024-03-04T16:28:05.131 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.131 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005023550ebebe5a7c-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.134 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.134 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.134 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.134 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.135 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.135 DEBUG:boto:Data: 2024-03-04T16:28:05.135 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.135 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.135 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.135 DEBUG:boto:Params: {} 2024-03-04T16:28:05.135 DEBUG:boto:Token: None 2024-03-04T16:28:05.135 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F/x9fxI/dt38cDFfrcV4XMONsPE= 2024-03-04T16:28:05.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F/x9fxI/dt38cDFfrcV4XMONsPE=', 'Content-Length': '0'} 2024-03-04T16:28:05.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035465bf16763e240-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.138 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.138 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.138 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.138 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.139 DEBUG:boto:Data: 2024-03-04T16:28:05.139 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.139 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.139 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.139 DEBUG:boto:Params: {} 2024-03-04T16:28:05.139 DEBUG:boto:Token: None 2024-03-04T16:28:05.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.139 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F/x9fxI/dt38cDFfrcV4XMONsPE= 2024-03-04T16:28:05.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F/x9fxI/dt38cDFfrcV4XMONsPE=', 'Content-Length': '0'} 2024-03-04T16:28:05.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c7f5dd9e9da1b15-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:05.142 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.142 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.142 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.142 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.142 DEBUG:boto:Method: GET 2024-03-04T16:28:05.142 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.142 DEBUG:boto:Data: 2024-03-04T16:28:05.142 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.142 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.142 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.143 DEBUG:boto:Params: {} 2024-03-04T16:28:05.143 DEBUG:boto:Token: None 2024-03-04T16:28:05.143 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.143 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be4f75692c0fbd65-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.145 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.145 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.145 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.145 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.145 DEBUG:boto:Method: GET 2024-03-04T16:28:05.145 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.145 DEBUG:boto:Data: 2024-03-04T16:28:05.145 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.145 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.146 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.146 DEBUG:boto:Params: {} 2024-03-04T16:28:05.146 DEBUG:boto:Token: None 2024-03-04T16:28:05.146 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.146 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091fac20e03cba353-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.192 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.192 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.192 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.192 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.193 DEBUG:boto:Method: GET 2024-03-04T16:28:05.193 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.193 DEBUG:boto:Data: 2024-03-04T16:28:05.193 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.193 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.193 DEBUG:boto:Params: {} 2024-03-04T16:28:05.193 DEBUG:boto:Token: None 2024-03-04T16:28:05.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a1aa8d5301a6aa4-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.196 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.196 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.197 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.197 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.197 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.197 DEBUG:boto:Method: GET 2024-03-04T16:28:05.197 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:05.197 DEBUG:boto:Data: 2024-03-04T16:28:05.197 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:05.197 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.197 DEBUG:boto:Params: {} 2024-03-04T16:28:05.198 DEBUG:boto:Token: None 2024-03-04T16:28:05.198 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f5a8c20c5ebb63-0065e5f695-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.200 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.201 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone3, test-zone1} 2024-03-04T16:28:05.201 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-19 zones={test-zone3, test-zone2} 2024-03-04T16:28:05.201 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.201 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.201 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.201 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:05.201 DEBUG:boto:Data: 2024-03-04T16:28:05.201 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.201 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.202 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.202 DEBUG:boto:Params: {} 2024-03-04T16:28:05.202 DEBUG:boto:Token: None 2024-03-04T16:28:05.202 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qFYGZSL8hcmYv/FPEvOeIMXKjwU= 2024-03-04T16:28:05.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qFYGZSL8hcmYv/FPEvOeIMXKjwU=', 'Content-Length': '0'} 2024-03-04T16:28:05.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010560c5c41333aa5-0065e5f695-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.204 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.204 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.204 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.204 DEBUG:boto:Path: /sbnvju-19/ 2024-03-04T16:28:05.204 DEBUG:boto:Data: 2024-03-04T16:28:05.205 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.205 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.205 DEBUG:boto:Params: {} 2024-03-04T16:28:05.205 DEBUG:boto:Token: None 2024-03-04T16:28:05.205 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qFYGZSL8hcmYv/FPEvOeIMXKjwU= 2024-03-04T16:28:05.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qFYGZSL8hcmYv/FPEvOeIMXKjwU=', 'Content-Length': '0'} 2024-03-04T16:28:05.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab7a354c1a9c090c-0065e5f695-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.208 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:05.208 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.208 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.208 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.208 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.209 DEBUG:boto:Method: GET 2024-03-04T16:28:05.209 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.209 DEBUG:boto:Data: 2024-03-04T16:28:05.209 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.209 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.209 DEBUG:boto:Params: {} 2024-03-04T16:28:05.209 DEBUG:boto:Token: None 2024-03-04T16:28:05.209 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/?versions 2024-03-04T16:28:05.209 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI= 2024-03-04T16:28:05.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI=', 'Content-Length': '0'} 2024-03-04T16:28:05.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9b43d1f696625c-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.212 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:05.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:05.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:05.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:05.213 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:05.213 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.214 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.214 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.214 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.214 DEBUG:boto:Method: GET 2024-03-04T16:28:05.214 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.214 DEBUG:boto:Data: 2024-03-04T16:28:05.214 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.214 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.214 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.214 DEBUG:boto:Params: {} 2024-03-04T16:28:05.214 DEBUG:boto:Token: None 2024-03-04T16:28:05.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/?versions 2024-03-04T16:28:05.215 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI= 2024-03-04T16:28:05.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI=', 'Content-Length': '0'} 2024-03-04T16:28:05.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee2e94f8afcc90d-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.217 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.218 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:05.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:05.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:05.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:05.218 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.218 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.218 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.218 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.219 DEBUG:boto:Method: GET 2024-03-04T16:28:05.219 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.219 DEBUG:boto:Data: 2024-03-04T16:28:05.219 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.219 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.219 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.219 DEBUG:boto:Params: {} 2024-03-04T16:28:05.219 DEBUG:boto:Token: None 2024-03-04T16:28:05.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/?versions 2024-03-04T16:28:05.219 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI= 2024-03-04T16:28:05.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI=', 'Content-Length': '0'} 2024-03-04T16:28:05.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc6b0de53f4bfcc-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.222 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.222 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.222 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.222 DEBUG:boto:path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.222 DEBUG:boto:auth_path=/sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.223 DEBUG:boto:Method: GET 2024-03-04T16:28:05.223 DEBUG:boto:Path: /sbnvju-19/?versions&max-keys=999 2024-03-04T16:28:05.223 DEBUG:boto:Data: 2024-03-04T16:28:05.223 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.223 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.223 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.223 DEBUG:boto:Params: {} 2024-03-04T16:28:05.223 DEBUG:boto:Token: None 2024-03-04T16:28:05.223 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/?versions 2024-03-04T16:28:05.223 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI= 2024-03-04T16:28:05.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0ZS4mfBhRn/bcwDuHIq66CCG1oI=', 'Content-Length': '0'} 2024-03-04T16:28:05.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051805da9bf0a6b56-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.226 DEBUG:boto:b'sbnvju-19999falsedummy-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2true2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone19t1mJz7w9pip0P8QyFarlLmpHfhi5LWtrue2024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2xbDXhMB8gOhPmHgCI8V9BaruC8RlINHtrue2024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3dKrXOf612ybfuQHfE5bg4qwg1H5ZEzltrue2024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:05.227 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.227 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.227 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.227 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.227 DEBUG:boto:Method: GET 2024-03-04T16:28:05.227 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.227 DEBUG:boto:Data: 2024-03-04T16:28:05.227 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.227 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.227 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.228 DEBUG:boto:Params: {} 2024-03-04T16:28:05.228 DEBUG:boto:Token: None 2024-03-04T16:28:05.228 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.228 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be12a3417af08596-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.230 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.230 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.230 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.231 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.231 DEBUG:boto:Method: GET 2024-03-04T16:28:05.231 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.231 DEBUG:boto:Data: 2024-03-04T16:28:05.231 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.231 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.231 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.231 DEBUG:boto:Params: {} 2024-03-04T16:28:05.231 DEBUG:boto:Token: None 2024-03-04T16:28:05.231 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.232 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004902189c2d344293-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.234 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.234 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.234 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.234 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.234 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.234 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.234 DEBUG:boto:Data: 2024-03-04T16:28:05.234 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.234 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.235 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.235 DEBUG:boto:Params: {} 2024-03-04T16:28:05.235 DEBUG:boto:Token: None 2024-03-04T16:28:05.235 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.235 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mCJlYRqaFABaClvWW7zbMMxEdxo= 2024-03-04T16:28:05.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mCJlYRqaFABaClvWW7zbMMxEdxo=', 'Content-Length': '0'} 2024-03-04T16:28:05.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b95ef2e4abc03087-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.237 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.238 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.238 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.238 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.238 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.238 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.238 DEBUG:boto:Data: 2024-03-04T16:28:05.238 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.238 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.239 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.239 DEBUG:boto:Params: {} 2024-03-04T16:28:05.239 DEBUG:boto:Token: None 2024-03-04T16:28:05.239 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.240 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mCJlYRqaFABaClvWW7zbMMxEdxo= 2024-03-04T16:28:05.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mCJlYRqaFABaClvWW7zbMMxEdxo=', 'Content-Length': '0'} 2024-03-04T16:28:05.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057b670e3b682ae79-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:05.242 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.242 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.242 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.242 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.243 DEBUG:boto:Method: GET 2024-03-04T16:28:05.243 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.243 DEBUG:boto:Data: 2024-03-04T16:28:05.243 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.243 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.243 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.243 DEBUG:boto:Params: {} 2024-03-04T16:28:05.243 DEBUG:boto:Token: None 2024-03-04T16:28:05.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.244 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a78ecb1001329baf-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.246 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.246 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.246 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.246 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.246 DEBUG:boto:Method: GET 2024-03-04T16:28:05.246 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.246 DEBUG:boto:Data: 2024-03-04T16:28:05.246 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.247 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.247 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.247 DEBUG:boto:Params: {} 2024-03-04T16:28:05.247 DEBUG:boto:Token: None 2024-03-04T16:28:05.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.247 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e46844cb617c117-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.249 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.250 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.250 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.250 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.250 DEBUG:boto:Method: GET 2024-03-04T16:28:05.250 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.250 DEBUG:boto:Data: 2024-03-04T16:28:05.250 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.250 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.250 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.250 DEBUG:boto:Params: {} 2024-03-04T16:28:05.250 DEBUG:boto:Token: None 2024-03-04T16:28:05.251 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.251 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312116d0223dbdd0-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.296 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.297 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.297 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.297 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.297 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.297 DEBUG:boto:Method: GET 2024-03-04T16:28:05.297 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.297 DEBUG:boto:Data: 2024-03-04T16:28:05.297 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.298 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.298 DEBUG:boto:Params: {} 2024-03-04T16:28:05.298 DEBUG:boto:Token: None 2024-03-04T16:28:05.298 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687d1ec83aecf13b-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.301 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.302 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.302 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.302 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.302 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:05.302 DEBUG:boto:Data: 2024-03-04T16:28:05.302 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.302 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.302 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.302 DEBUG:boto:Params: {} 2024-03-04T16:28:05.302 DEBUG:boto:Token: None 2024-03-04T16:28:05.303 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy 2024-03-04T16:28:05.303 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bUw7QBEPmq0LPtdJb76ejvWxdJU= 2024-03-04T16:28:05.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bUw7QBEPmq0LPtdJb76ejvWxdJU=', 'Content-Length': '0'} 2024-03-04T16:28:05.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008393815283d2ebda-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.305 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.305 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.305 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.305 DEBUG:boto:Path: /sbnvju-19/dummy 2024-03-04T16:28:05.305 DEBUG:boto:Data: 2024-03-04T16:28:05.305 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.306 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.306 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.306 DEBUG:boto:Params: {} 2024-03-04T16:28:05.306 DEBUG:boto:Token: None 2024-03-04T16:28:05.306 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy 2024-03-04T16:28:05.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bUw7QBEPmq0LPtdJb76ejvWxdJU= 2024-03-04T16:28:05.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bUw7QBEPmq0LPtdJb76ejvWxdJU=', 'Content-Length': '0'} 2024-03-04T16:28:05.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008dac379014391c7c-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:05.309 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.309 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.309 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.309 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.309 DEBUG:boto:Method: GET 2024-03-04T16:28:05.309 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.310 DEBUG:boto:Data: 2024-03-04T16:28:05.310 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.310 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.310 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.310 DEBUG:boto:Params: {} 2024-03-04T16:28:05.310 DEBUG:boto:Token: None 2024-03-04T16:28:05.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003facbf47cdd1e76c-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.312 DEBUG:boto:path=/sbnvju-19/dummy 2024-03-04T16:28:05.313 DEBUG:boto:auth_path=/sbnvju-19/dummy 2024-03-04T16:28:05.313 DEBUG:boto:path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.313 DEBUG:boto:auth_path=/sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.313 DEBUG:boto:Method: GET 2024-03-04T16:28:05.313 DEBUG:boto:Path: /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.313 DEBUG:boto:Data: 2024-03-04T16:28:05.313 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.313 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.313 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.313 DEBUG:boto:Params: {} 2024-03-04T16:28:05.313 DEBUG:boto:Token: None 2024-03-04T16:28:05.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/dummy?versionId=-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2 2024-03-04T16:28:05.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY= 2024-03-04T16:28:05.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wksGbf9crfJR9jOLyFdrWM4oAvY=', 'Content-Length': '0'} 2024-03-04T16:28:05.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:14 GMT'), ('x-amz-version-id', '-KwFHnWpB3uKLBIiNeYNpq3XwIuMAw2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077011c1ff7b8438f-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.316 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.316 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.316 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.316 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.316 DEBUG:boto:Method: GET 2024-03-04T16:28:05.316 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.316 DEBUG:boto:Data: 2024-03-04T16:28:05.316 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.317 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.317 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.317 DEBUG:boto:Params: {} 2024-03-04T16:28:05.317 DEBUG:boto:Token: None 2024-03-04T16:28:05.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.317 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba71268e4b72dd1f-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.364 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.365 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.365 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.365 DEBUG:boto:path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.365 DEBUG:boto:auth_path=/sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.365 DEBUG:boto:Method: GET 2024-03-04T16:28:05.365 DEBUG:boto:Path: /sbnvju-19/?prefix=dummy 2024-03-04T16:28:05.365 DEBUG:boto:Data: 2024-03-04T16:28:05.366 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.366 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.366 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.366 DEBUG:boto:Params: {} 2024-03-04T16:28:05.366 DEBUG:boto:Token: None 2024-03-04T16:28:05.366 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041d8add669980bfe-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.369 DEBUG:boto:b'sbnvju-19dummy1000falsedummy2024-03-04T16:24:14.917Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:05.370 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.370 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.370 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.370 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.370 DEBUG:boto:Method: GET 2024-03-04T16:28:05.370 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.370 DEBUG:boto:Data: 2024-03-04T16:28:05.370 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.370 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.370 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.370 DEBUG:boto:Params: {} 2024-03-04T16:28:05.372 DEBUG:boto:Token: None 2024-03-04T16:28:05.373 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a10c891ff7f095d-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.416 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.417 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.417 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.417 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.417 DEBUG:boto:Method: GET 2024-03-04T16:28:05.417 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.417 DEBUG:boto:Data: 2024-03-04T16:28:05.417 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.417 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.417 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.417 DEBUG:boto:Params: {} 2024-03-04T16:28:05.417 DEBUG:boto:Token: None 2024-03-04T16:28:05.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8d30d8becf3593d-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.420 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.420 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.420 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.420 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.421 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.421 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.421 DEBUG:boto:Data: 2024-03-04T16:28:05.421 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.421 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.421 DEBUG:boto:Params: {} 2024-03-04T16:28:05.421 DEBUG:boto:Token: None 2024-03-04T16:28:05.421 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sCIbC35V2lEiYVZLTa025b+74bw= 2024-03-04T16:28:05.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sCIbC35V2lEiYVZLTa025b+74bw=', 'Content-Length': '0'} 2024-03-04T16:28:05.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e2f541991b91522-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.424 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.424 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.424 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.424 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.424 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.424 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.424 DEBUG:boto:Data: 2024-03-04T16:28:05.424 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.424 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.424 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.424 DEBUG:boto:Params: {} 2024-03-04T16:28:05.425 DEBUG:boto:Token: None 2024-03-04T16:28:05.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sCIbC35V2lEiYVZLTa025b+74bw= 2024-03-04T16:28:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sCIbC35V2lEiYVZLTa025b+74bw=', 'Content-Length': '0'} 2024-03-04T16:28:05.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000257c8e5d12db67a1-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:05.428 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.428 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.428 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.428 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.428 DEBUG:boto:Method: GET 2024-03-04T16:28:05.428 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.428 DEBUG:boto:Data: 2024-03-04T16:28:05.428 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.428 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.428 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.429 DEBUG:boto:Params: {} 2024-03-04T16:28:05.429 DEBUG:boto:Token: None 2024-03-04T16:28:05.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000936485168b2b5ebb-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.472 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.472 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.472 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.472 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.473 DEBUG:boto:Method: GET 2024-03-04T16:28:05.473 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.473 DEBUG:boto:Data: 2024-03-04T16:28:05.473 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.473 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.473 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.473 DEBUG:boto:Params: {} 2024-03-04T16:28:05.473 DEBUG:boto:Token: None 2024-03-04T16:28:05.474 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdba4ecdae3207f1-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.516 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.516 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.517 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.517 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.517 DEBUG:boto:Method: GET 2024-03-04T16:28:05.517 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.517 DEBUG:boto:Data: 2024-03-04T16:28:05.517 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.518 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.518 DEBUG:boto:Params: {} 2024-03-04T16:28:05.518 DEBUG:boto:Token: None 2024-03-04T16:28:05.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce23a7b9d54c84f-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.564 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.565 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.565 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.566 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.566 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.566 DEBUG:boto:Method: GET 2024-03-04T16:28:05.566 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.566 DEBUG:boto:Data: 2024-03-04T16:28:05.566 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.566 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.566 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.566 DEBUG:boto:Params: {} 2024-03-04T16:28:05.567 DEBUG:boto:Token: None 2024-03-04T16:28:05.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5970f51ca8531c0-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.570 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.571 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.572 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.572 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.572 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.572 DEBUG:boto:Data: 2024-03-04T16:28:05.572 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.572 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.572 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.573 DEBUG:boto:Params: {} 2024-03-04T16:28:05.573 DEBUG:boto:Token: None 2024-03-04T16:28:05.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QSwvF96D0jfVSxWUOjCGgp1F/fs= 2024-03-04T16:28:05.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QSwvF96D0jfVSxWUOjCGgp1F/fs=', 'Content-Length': '0'} 2024-03-04T16:28:05.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000856d6ee491634ff4-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.576 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.577 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.577 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.577 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.577 DEBUG:boto:Data: 2024-03-04T16:28:05.577 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.577 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.577 DEBUG:boto:Params: {} 2024-03-04T16:28:05.578 DEBUG:boto:Token: None 2024-03-04T16:28:05.579 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.579 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QSwvF96D0jfVSxWUOjCGgp1F/fs= 2024-03-04T16:28:05.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QSwvF96D0jfVSxWUOjCGgp1F/fs=', 'Content-Length': '0'} 2024-03-04T16:28:05.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf216e94fb439b05-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:05.582 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.582 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.582 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.582 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.583 DEBUG:boto:Method: GET 2024-03-04T16:28:05.583 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.583 DEBUG:boto:Data: 2024-03-04T16:28:05.583 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.583 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.583 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.583 DEBUG:boto:Params: {} 2024-03-04T16:28:05.583 DEBUG:boto:Token: None 2024-03-04T16:28:05.583 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c7fdb104a0f3390-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.628 DEBUG:boto:path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.628 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1 2024-03-04T16:28:05.628 DEBUG:boto:path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.629 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.629 DEBUG:boto:Method: GET 2024-03-04T16:28:05.629 DEBUG:boto:Path: /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.629 DEBUG:boto:Data: 2024-03-04T16:28:05.629 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.629 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.629 DEBUG:boto:Params: {} 2024-03-04T16:28:05.629 DEBUG:boto:Token: None 2024-03-04T16:28:05.629 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone1?versionId=9t1mJz7w9pip0P8QyFarlLmpHfhi5LW 2024-03-04T16:28:05.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ= 2024-03-04T16:28:05.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SebfzN4uTUDITdEFJnf159e6GSQ=', 'Content-Length': '0'} 2024-03-04T16:28:05.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', '9t1mJz7w9pip0P8QyFarlLmpHfhi5LW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000455a284eaeb11fec-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.672 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.672 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.672 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.673 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.673 DEBUG:boto:Method: GET 2024-03-04T16:28:05.673 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.673 DEBUG:boto:Data: 2024-03-04T16:28:05.673 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.673 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.673 DEBUG:boto:Params: {} 2024-03-04T16:28:05.673 DEBUG:boto:Token: None 2024-03-04T16:28:05.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f9d502fe3b7560-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.724 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.725 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.725 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.725 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.725 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.725 DEBUG:boto:Method: GET 2024-03-04T16:28:05.725 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone1 2024-03-04T16:28:05.725 DEBUG:boto:Data: 2024-03-04T16:28:05.725 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.726 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.726 DEBUG:boto:Params: {} 2024-03-04T16:28:05.726 DEBUG:boto:Token: None 2024-03-04T16:28:05.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b3c9a8e62d41f4-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.729 DEBUG:boto:b'sbnvju-19obj-test-zone11000falseobj-test-zone12024-03-04T16:25:35.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:05.730 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.730 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.730 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.730 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.730 DEBUG:boto:Method: GET 2024-03-04T16:28:05.730 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.730 DEBUG:boto:Data: 2024-03-04T16:28:05.730 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.731 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.731 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.731 DEBUG:boto:Params: {} 2024-03-04T16:28:05.731 DEBUG:boto:Token: None 2024-03-04T16:28:05.731 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.731 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064001ce676d3fcf2-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.734 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.734 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.734 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.734 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.734 DEBUG:boto:Method: GET 2024-03-04T16:28:05.734 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.735 DEBUG:boto:Data: 2024-03-04T16:28:05.735 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.735 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.735 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.735 DEBUG:boto:Params: {} 2024-03-04T16:28:05.735 DEBUG:boto:Token: None 2024-03-04T16:28:05.735 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.735 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000566600c3abe1e3e0-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.738 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.738 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.738 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.738 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.738 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.738 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.738 DEBUG:boto:Data: 2024-03-04T16:28:05.739 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.739 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.739 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.739 DEBUG:boto:Params: {} 2024-03-04T16:28:05.739 DEBUG:boto:Token: None 2024-03-04T16:28:05.739 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.739 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/8wUFTkEz/sHcrpuy1QBls/XSRo= 2024-03-04T16:28:05.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/8wUFTkEz/sHcrpuy1QBls/XSRo=', 'Content-Length': '0'} 2024-03-04T16:28:05.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099db5ea4bada340e-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.741 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.742 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.742 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.742 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.742 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.742 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.742 DEBUG:boto:Data: 2024-03-04T16:28:05.742 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.742 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.742 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.742 DEBUG:boto:Params: {} 2024-03-04T16:28:05.742 DEBUG:boto:Token: None 2024-03-04T16:28:05.743 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.743 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/8wUFTkEz/sHcrpuy1QBls/XSRo= 2024-03-04T16:28:05.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/8wUFTkEz/sHcrpuy1QBls/XSRo=', 'Content-Length': '0'} 2024-03-04T16:28:05.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f3b7d61f85ccc9b-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:05.745 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.745 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.745 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.745 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.745 DEBUG:boto:Method: GET 2024-03-04T16:28:05.745 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.745 DEBUG:boto:Data: 2024-03-04T16:28:05.745 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.745 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.745 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.746 DEBUG:boto:Params: {} 2024-03-04T16:28:05.746 DEBUG:boto:Token: None 2024-03-04T16:28:05.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac189c1a57336e5-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.796 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.797 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.797 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.797 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.797 DEBUG:boto:Method: GET 2024-03-04T16:28:05.797 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.797 DEBUG:boto:Data: 2024-03-04T16:28:05.797 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.797 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.797 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.798 DEBUG:boto:Params: {} 2024-03-04T16:28:05.798 DEBUG:boto:Token: None 2024-03-04T16:28:05.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e89947e6ac1d7cd9-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.801 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.801 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.801 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.801 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.801 DEBUG:boto:Method: GET 2024-03-04T16:28:05.801 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.802 DEBUG:boto:Data: 2024-03-04T16:28:05.802 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.802 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.802 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.802 DEBUG:boto:Params: {} 2024-03-04T16:28:05.802 DEBUG:boto:Token: None 2024-03-04T16:28:05.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.803 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031fca2963a3ab993-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.848 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.849 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.849 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.849 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.849 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.849 DEBUG:boto:Method: GET 2024-03-04T16:28:05.849 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.849 DEBUG:boto:Data: 2024-03-04T16:28:05.849 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.850 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.850 DEBUG:boto:Params: {} 2024-03-04T16:28:05.850 DEBUG:boto:Token: None 2024-03-04T16:28:05.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c83b6d3c4a2cb2f-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.853 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.854 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.854 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.854 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.854 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.854 DEBUG:boto:Data: 2024-03-04T16:28:05.854 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.854 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.854 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.855 DEBUG:boto:Params: {} 2024-03-04T16:28:05.855 DEBUG:boto:Token: None 2024-03-04T16:28:05.855 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0C9rJj7F9h3K3NZ/BzReSrOCit0= 2024-03-04T16:28:05.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0C9rJj7F9h3K3NZ/BzReSrOCit0=', 'Content-Length': '0'} 2024-03-04T16:28:05.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa58c381c9b7dc81-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.858 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.858 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.859 DEBUG:boto:Method: HEAD 2024-03-04T16:28:05.859 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.859 DEBUG:boto:Data: 2024-03-04T16:28:05.859 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.859 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.859 DEBUG:boto:Params: {} 2024-03-04T16:28:05.859 DEBUG:boto:Token: None 2024-03-04T16:28:05.859 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0C9rJj7F9h3K3NZ/BzReSrOCit0= 2024-03-04T16:28:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0C9rJj7F9h3K3NZ/BzReSrOCit0=', 'Content-Length': '0'} 2024-03-04T16:28:05.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5b674e8a4943068-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:05.862 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.863 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.863 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.863 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.863 DEBUG:boto:Method: GET 2024-03-04T16:28:05.863 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.863 DEBUG:boto:Data: 2024-03-04T16:28:05.863 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.863 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.863 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.863 DEBUG:boto:Params: {} 2024-03-04T16:28:05.863 DEBUG:boto:Token: None 2024-03-04T16:28:05.864 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.864 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000725c8c886aac3681-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.908 DEBUG:boto:path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.908 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2 2024-03-04T16:28:05.908 DEBUG:boto:path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.908 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.909 DEBUG:boto:Method: GET 2024-03-04T16:28:05.909 DEBUG:boto:Path: /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.909 DEBUG:boto:Data: 2024-03-04T16:28:05.909 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.909 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.909 DEBUG:boto:Params: {} 2024-03-04T16:28:05.909 DEBUG:boto:Token: None 2024-03-04T16:28:05.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone2?versionId=xbDXhMB8gOhPmHgCI8V9BaruC8RlINH 2024-03-04T16:28:05.909 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs= 2024-03-04T16:28:05.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R0j5lTyx4idLgoAUQvSHZIfIRjs=', 'Content-Length': '0'} 2024-03-04T16:28:05.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:35 GMT'), ('x-amz-version-id', 'xbDXhMB8gOhPmHgCI8V9BaruC8RlINH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d1c69d834612410-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.912 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.912 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.912 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.912 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.913 DEBUG:boto:Method: GET 2024-03-04T16:28:05.913 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.913 DEBUG:boto:Data: 2024-03-04T16:28:05.913 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.913 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.913 DEBUG:boto:Params: {} 2024-03-04T16:28:05.913 DEBUG:boto:Token: None 2024-03-04T16:28:05.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade84b0fa391608d-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.960 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.961 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:05.961 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:05.961 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.961 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.961 DEBUG:boto:Method: GET 2024-03-04T16:28:05.961 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone2 2024-03-04T16:28:05.961 DEBUG:boto:Data: 2024-03-04T16:28:05.961 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.961 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.961 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.962 DEBUG:boto:Params: {} 2024-03-04T16:28:05.962 DEBUG:boto:Token: None 2024-03-04T16:28:05.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/ 2024-03-04T16:28:05.962 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E= 2024-03-04T16:28:05.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W/qD8hfT/SMe0Pqk4g3SSkytr4E=', 'Content-Length': '0'} 2024-03-04T16:28:05.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d529b4cff60350-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.965 DEBUG:boto:b'sbnvju-19obj-test-zone21000falseobj-test-zone22024-03-04T16:25:35.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:05.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:05.965 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.965 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.965 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.965 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.965 DEBUG:boto:Method: GET 2024-03-04T16:28:05.966 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.966 DEBUG:boto:Data: 2024-03-04T16:28:05.966 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:05.966 DEBUG:boto:Port: 8001 2024-03-04T16:28:05.966 DEBUG:boto:Params: {} 2024-03-04T16:28:05.966 DEBUG:boto:Token: None 2024-03-04T16:28:05.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3b2dc4cd8728f4e-0065e5f695-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:05.972 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.972 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:05.972 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.973 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.973 DEBUG:boto:Method: GET 2024-03-04T16:28:05.973 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.973 DEBUG:boto:Data: 2024-03-04T16:28:05.973 DEBUG:boto:Headers: {} 2024-03-04T16:28:05.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:05.973 DEBUG:boto:Port: 8000 2024-03-04T16:28:05.973 DEBUG:boto:Params: {} 2024-03-04T16:28:05.973 DEBUG:boto:Token: None 2024-03-04T16:28:05.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:05 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:05.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag= 2024-03-04T16:28:05.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:05 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ia6mha49+aEipWas9lH9Y1pfYag=', 'Content-Length': '0'} 2024-03-04T16:28:05.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052df109099f616de-0065e5f695-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:05 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.020 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.021 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.021 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.021 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.021 DEBUG:boto:Method: HEAD 2024-03-04T16:28:06.021 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.021 DEBUG:boto:Data: 2024-03-04T16:28:06.022 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.022 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.022 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.022 DEBUG:boto:Params: {} 2024-03-04T16:28:06.022 DEBUG:boto:Token: None 2024-03-04T16:28:06.023 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.023 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0FESAg/QpXvGsNu0jbr70TY5258= 2024-03-04T16:28:06.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0FESAg/QpXvGsNu0jbr70TY5258=', 'Content-Length': '0'} 2024-03-04T16:28:06.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e304793ee97da95-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.026 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.027 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.027 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.027 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.029 DEBUG:boto:Method: HEAD 2024-03-04T16:28:06.029 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.029 DEBUG:boto:Data: 2024-03-04T16:28:06.029 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.030 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.030 DEBUG:boto:Params: {} 2024-03-04T16:28:06.030 DEBUG:boto:Token: None 2024-03-04T16:28:06.030 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0FESAg/QpXvGsNu0jbr70TY5258= 2024-03-04T16:28:06.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0FESAg/QpXvGsNu0jbr70TY5258=', 'Content-Length': '0'} 2024-03-04T16:28:06.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024b5255bce7b47e5-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:06.032 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.033 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.033 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.033 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.033 DEBUG:boto:Method: GET 2024-03-04T16:28:06.033 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.033 DEBUG:boto:Data: 2024-03-04T16:28:06.033 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.033 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.033 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.033 DEBUG:boto:Params: {} 2024-03-04T16:28:06.033 DEBUG:boto:Token: None 2024-03-04T16:28:06.034 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0= 2024-03-04T16:28:06.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0=', 'Content-Length': '0'} 2024-03-04T16:28:06.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f235976d7008575-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.036 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.037 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.037 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.037 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.037 DEBUG:boto:Method: GET 2024-03-04T16:28:06.037 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.037 DEBUG:boto:Data: 2024-03-04T16:28:06.037 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.038 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.038 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.038 DEBUG:boto:Params: {} 2024-03-04T16:28:06.038 DEBUG:boto:Token: None 2024-03-04T16:28:06.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.039 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0= 2024-03-04T16:28:06.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0=', 'Content-Length': '0'} 2024-03-04T16:28:06.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c02e3fd9b80e16fc-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.088 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:06.088 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:06.088 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.089 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.089 DEBUG:boto:Method: GET 2024-03-04T16:28:06.089 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.089 DEBUG:boto:Data: 2024-03-04T16:28:06.089 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.089 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.089 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.089 DEBUG:boto:Params: {} 2024-03-04T16:28:06.089 DEBUG:boto:Token: None 2024-03-04T16:28:06.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/ 2024-03-04T16:28:06.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM= 2024-03-04T16:28:06.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM=', 'Content-Length': '0'} 2024-03-04T16:28:06.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2945819683746b-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.093 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:06.093 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:06.094 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:06.094 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.094 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.094 DEBUG:boto:Method: GET 2024-03-04T16:28:06.094 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.094 DEBUG:boto:Data: 2024-03-04T16:28:06.094 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.094 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.094 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.094 DEBUG:boto:Params: {} 2024-03-04T16:28:06.095 DEBUG:boto:Token: None 2024-03-04T16:28:06.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/ 2024-03-04T16:28:06.095 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM= 2024-03-04T16:28:06.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM=', 'Content-Length': '0'} 2024-03-04T16:28:06.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a470a1d5ab61be13-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.099 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:06.100 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.100 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.100 DEBUG:boto:Method: HEAD 2024-03-04T16:28:06.100 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.100 DEBUG:boto:Data: 2024-03-04T16:28:06.100 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.100 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.100 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.100 DEBUG:boto:Params: {} 2024-03-04T16:28:06.101 DEBUG:boto:Token: None 2024-03-04T16:28:06.101 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.101 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7fG0ZY1/fK8bxxtwKBKKzF1cvc= 2024-03-04T16:28:06.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7fG0ZY1/fK8bxxtwKBKKzF1cvc=', 'Content-Length': '0'} 2024-03-04T16:28:06.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000701993b5fdfc8fd6-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.104 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.104 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.104 DEBUG:boto:Method: HEAD 2024-03-04T16:28:06.104 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.104 DEBUG:boto:Data: 2024-03-04T16:28:06.104 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.104 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.105 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.105 DEBUG:boto:Params: {} 2024-03-04T16:28:06.105 DEBUG:boto:Token: None 2024-03-04T16:28:06.105 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.105 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7fG0ZY1/fK8bxxtwKBKKzF1cvc= 2024-03-04T16:28:06.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7fG0ZY1/fK8bxxtwKBKKzF1cvc=', 'Content-Length': '0'} 2024-03-04T16:28:06.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af0feaa7bfb04bac-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:06.108 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.108 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.109 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.109 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.109 DEBUG:boto:Method: GET 2024-03-04T16:28:06.109 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.109 DEBUG:boto:Data: 2024-03-04T16:28:06.109 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.109 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.109 DEBUG:boto:Params: {} 2024-03-04T16:28:06.110 DEBUG:boto:Token: None 2024-03-04T16:28:06.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0= 2024-03-04T16:28:06.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0=', 'Content-Length': '0'} 2024-03-04T16:28:06.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000894f49314361938c-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.160 DEBUG:boto:path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.160 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3 2024-03-04T16:28:06.160 DEBUG:boto:path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.160 DEBUG:boto:auth_path=/sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.160 DEBUG:boto:Method: GET 2024-03-04T16:28:06.161 DEBUG:boto:Path: /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.161 DEBUG:boto:Data: 2024-03-04T16:28:06.161 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.161 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.161 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.161 DEBUG:boto:Params: {} 2024-03-04T16:28:06.161 DEBUG:boto:Token: None 2024-03-04T16:28:06.161 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/obj-test-zone3?versionId=dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl 2024-03-04T16:28:06.161 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0= 2024-03-04T16:28:06.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4apqr2CTzk6PJV/84VQzg2nNut0=', 'Content-Length': '0'} 2024-03-04T16:28:06.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'dKrXOf612ybfuQHfE5bg4qwg1H5ZEzl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-19:fc294220-5038-4f25-aeee-cac2c5838729.4341.28'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab06dc65f2c6a41a-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.205 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:06.205 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:06.205 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.206 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.206 DEBUG:boto:Method: GET 2024-03-04T16:28:06.206 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.206 DEBUG:boto:Data: 2024-03-04T16:28:06.207 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.207 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:06.207 DEBUG:boto:Port: 8001 2024-03-04T16:28:06.207 DEBUG:boto:Params: {} 2024-03-04T16:28:06.208 DEBUG:boto:Token: None 2024-03-04T16:28:06.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/ 2024-03-04T16:28:06.208 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM= 2024-03-04T16:28:06.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM=', 'Content-Length': '0'} 2024-03-04T16:28:06.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea4def5d9c20ed8-0065e5f696-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.212 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:06.213 DEBUG:boto:path=/sbnvju-19/ 2024-03-04T16:28:06.213 DEBUG:boto:auth_path=/sbnvju-19/ 2024-03-04T16:28:06.214 DEBUG:boto:path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.214 DEBUG:boto:auth_path=/sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.214 DEBUG:boto:Method: GET 2024-03-04T16:28:06.214 DEBUG:boto:Path: /sbnvju-19/?prefix=obj-test-zone3 2024-03-04T16:28:06.215 DEBUG:boto:Data: 2024-03-04T16:28:06.216 DEBUG:boto:Headers: {} 2024-03-04T16:28:06.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:06.216 DEBUG:boto:Port: 8000 2024-03-04T16:28:06.216 DEBUG:boto:Params: {} 2024-03-04T16:28:06.217 DEBUG:boto:Token: None 2024-03-04T16:28:06.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:06 GMT /sbnvju-19/ 2024-03-04T16:28:06.217 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM= 2024-03-04T16:28:06.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rxLm+mXcXTY2j99FLMSH5E0XcaM=', 'Content-Length': '0'} 2024-03-04T16:28:06.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647f01858a7f706a-0065e5f696-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:06.268 DEBUG:boto:b'sbnvju-19obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:06.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-19 zones={test-zone3, test-zone2} 2024-03-04T16:28:06.269 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-20 2024-03-04T16:28:06.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.269 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:06.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:06.381+0000 7f3b84c88800 1 bucket sync caught up with source: 2024-03-04T16:28:06.381 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000011.39.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:28:06.381 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.39.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:28:06.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:06.381+0000 7f3b84c88800 0 bucket checkpoint complete 2024-03-04T16:28:06.400 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-20 2024-03-04T16:28:06.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.401 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.465 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.465 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.519+0000 7f7ab5131800 1 bucket sync caught up with source: 2024-03-04T16:28:06.522 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000011.39.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:28:06.522 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000011.39.6, , 00000000003.20.10, , 00000000007.36.10, , , , , ] 2024-03-04T16:28:06.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.519+0000 7f7ab5131800 0 bucket checkpoint complete 2024-03-04T16:28:06.538 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-20 2024-03-04T16:28:06.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.539 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.593 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.594 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.643+0000 7f022d224800 1 bucket sync caught up with source: 2024-03-04T16:28:06.644 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:28:06.644 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:28:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.643+0000 7f022d224800 0 bucket checkpoint complete 2024-03-04T16:28:06.662 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-20 2024-03-04T16:28:06.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.663 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.718 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.718 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.768+0000 7f8f6400f800 1 bucket sync caught up with source: 2024-03-04T16:28:06.769 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:28:06.769 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.22.10, , 00000000011.34.6, , 00000000007.49.10, , , , , ] 2024-03-04T16:28:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.768+0000 7f8f6400f800 0 bucket checkpoint complete 2024-03-04T16:28:06.787 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-20 2024-03-04T16:28:06.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.787 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.845 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.845 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:06.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.895+0000 7fbf41a8c800 1 bucket sync caught up with source: 2024-03-04T16:28:06.896 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.20.10, , 00000000003.21.10, , 00000000015.63.6, , , , , ] 2024-03-04T16:28:06.896 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.20.10, , 00000000003.21.10, , 00000000015.63.6, , , , , ] 2024-03-04T16:28:06.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:06.895+0000 7fbf41a8c800 0 bucket checkpoint complete 2024-03-04T16:28:06.913 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-20 2024-03-04T16:28:06.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:06.914 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:06.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:06.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:07.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:07.018+0000 7f5e59b33800 1 bucket sync caught up with source: 2024-03-04T16:28:07.018 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.20.10, , 00000000003.21.10, , 00000000015.63.6, , , , , ] 2024-03-04T16:28:07.018 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.20.10, , 00000000003.21.10, , 00000000015.63.6, , , , , ] 2024-03-04T16:28:07.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:07.018+0000 7f5e59b33800 0 bucket checkpoint complete 2024-03-04T16:28:07.036 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone2, test-zone1} 2024-03-04T16:28:07.036 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.036 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.036 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.036 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:07.036 DEBUG:boto:Data: 2024-03-04T16:28:07.036 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.036 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.036 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.037 DEBUG:boto:Params: {} 2024-03-04T16:28:07.037 DEBUG:boto:Token: None 2024-03-04T16:28:07.037 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.037 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4= 2024-03-04T16:28:07.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4=', 'Content-Length': '0'} 2024-03-04T16:28:07.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a7a69f24bd50f8-0065e5f697-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.039 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.040 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.040 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.040 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:07.040 DEBUG:boto:Data: 2024-03-04T16:28:07.040 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.040 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.040 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.040 DEBUG:boto:Params: {} 2024-03-04T16:28:07.040 DEBUG:boto:Token: None 2024-03-04T16:28:07.040 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.041 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4= 2024-03-04T16:28:07.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4=', 'Content-Length': '0'} 2024-03-04T16:28:07.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a79640015431bc60-0065e5f697-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:07.043 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.043 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.043 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.043 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.043 DEBUG:boto:Method: GET 2024-03-04T16:28:07.043 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.043 DEBUG:boto:Data: 2024-03-04T16:28:07.044 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.044 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.044 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.044 DEBUG:boto:Params: {} 2024-03-04T16:28:07.044 DEBUG:boto:Token: None 2024-03-04T16:28:07.044 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.044 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022871ec31fcd5414-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.047 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:07.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:07.048 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.048 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.048 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.048 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.048 DEBUG:boto:Method: GET 2024-03-04T16:28:07.048 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.048 DEBUG:boto:Data: 2024-03-04T16:28:07.048 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.048 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.048 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.048 DEBUG:boto:Params: {} 2024-03-04T16:28:07.049 DEBUG:boto:Token: None 2024-03-04T16:28:07.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039239df53e21075f-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.051 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:07.052 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.053 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.053 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.053 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.053 DEBUG:boto:Method: GET 2024-03-04T16:28:07.053 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.053 DEBUG:boto:Data: 2024-03-04T16:28:07.053 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.053 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.053 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.053 DEBUG:boto:Params: {} 2024-03-04T16:28:07.053 DEBUG:boto:Token: None 2024-03-04T16:28:07.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.054 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ebee2c0514ad16-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.056 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.056 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.057 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.057 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.057 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.057 DEBUG:boto:Method: GET 2024-03-04T16:28:07.057 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.057 DEBUG:boto:Data: 2024-03-04T16:28:07.057 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.057 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.057 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.057 DEBUG:boto:Params: {} 2024-03-04T16:28:07.057 DEBUG:boto:Token: None 2024-03-04T16:28:07.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f6fafba3d54631-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.060 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:07.061 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.061 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.061 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.061 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.061 DEBUG:boto:Method: GET 2024-03-04T16:28:07.061 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.061 DEBUG:boto:Data: 2024-03-04T16:28:07.061 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.061 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.061 DEBUG:boto:Params: {} 2024-03-04T16:28:07.062 DEBUG:boto:Token: None 2024-03-04T16:28:07.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000eccfd682cedc8e4-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.064 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.064 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.064 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.064 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.064 DEBUG:boto:Method: GET 2024-03-04T16:28:07.064 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.064 DEBUG:boto:Data: 2024-03-04T16:28:07.064 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.064 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.064 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.065 DEBUG:boto:Params: {} 2024-03-04T16:28:07.065 DEBUG:boto:Token: None 2024-03-04T16:28:07.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e617e8b23f00179a-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.069 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.069 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.069 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.069 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.069 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.070 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.070 DEBUG:boto:Data: 2024-03-04T16:28:07.070 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.070 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.070 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.070 DEBUG:boto:Params: {} 2024-03-04T16:28:07.070 DEBUG:boto:Token: None 2024-03-04T16:28:07.071 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.071 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g= 2024-03-04T16:28:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g=', 'Content-Length': '0'} 2024-03-04T16:28:07.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d3bfa63e4784f03-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.074 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.074 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.074 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.074 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.074 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.074 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.074 DEBUG:boto:Data: 2024-03-04T16:28:07.074 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.074 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.074 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.075 DEBUG:boto:Params: {} 2024-03-04T16:28:07.075 DEBUG:boto:Token: None 2024-03-04T16:28:07.075 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.075 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g= 2024-03-04T16:28:07.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g=', 'Content-Length': '0'} 2024-03-04T16:28:07.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c963b827be12d0fa-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:07.078 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.078 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.078 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.078 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.078 DEBUG:boto:Method: GET 2024-03-04T16:28:07.078 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.078 DEBUG:boto:Data: 2024-03-04T16:28:07.079 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.079 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.079 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.079 DEBUG:boto:Params: {} 2024-03-04T16:28:07.079 DEBUG:boto:Token: None 2024-03-04T16:28:07.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.079 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b85030584fee0d54-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.081 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.082 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.082 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.082 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.082 DEBUG:boto:Method: GET 2024-03-04T16:28:07.082 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.082 DEBUG:boto:Data: 2024-03-04T16:28:07.083 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.083 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.083 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.083 DEBUG:boto:Params: {} 2024-03-04T16:28:07.083 DEBUG:boto:Token: None 2024-03-04T16:28:07.083 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.083 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cf2e122a76a5c76-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.085 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.085 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.085 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.085 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.085 DEBUG:boto:Method: GET 2024-03-04T16:28:07.086 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.086 DEBUG:boto:Data: 2024-03-04T16:28:07.086 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.086 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.086 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.086 DEBUG:boto:Params: {} 2024-03-04T16:28:07.086 DEBUG:boto:Token: None 2024-03-04T16:28:07.086 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.087 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d2f25c7702cf07-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.089 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.090 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.090 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.090 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.090 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.090 DEBUG:boto:Method: GET 2024-03-04T16:28:07.090 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.090 DEBUG:boto:Data: 2024-03-04T16:28:07.091 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.091 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.091 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.091 DEBUG:boto:Params: {} 2024-03-04T16:28:07.091 DEBUG:boto:Token: None 2024-03-04T16:28:07.091 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.091 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a152ae41f2405d-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.140 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.140 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.141 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.141 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.141 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:07.141 DEBUG:boto:Data: 2024-03-04T16:28:07.141 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.141 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.141 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.141 DEBUG:boto:Params: {} 2024-03-04T16:28:07.141 DEBUG:boto:Token: None 2024-03-04T16:28:07.142 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy 2024-03-04T16:28:07.142 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Eg2/1GWBbPLfIEFfZMi42ujQgaY= 2024-03-04T16:28:07.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Eg2/1GWBbPLfIEFfZMi42ujQgaY=', 'Content-Length': '0'} 2024-03-04T16:28:07.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e18105ceb54812a2-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.145 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.145 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.145 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.145 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:07.145 DEBUG:boto:Data: 2024-03-04T16:28:07.146 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.146 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.146 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.146 DEBUG:boto:Params: {} 2024-03-04T16:28:07.146 DEBUG:boto:Token: None 2024-03-04T16:28:07.147 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy 2024-03-04T16:28:07.147 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Eg2/1GWBbPLfIEFfZMi42ujQgaY= 2024-03-04T16:28:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Eg2/1GWBbPLfIEFfZMi42ujQgaY=', 'Content-Length': '0'} 2024-03-04T16:28:07.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003928f6727bf49286-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:07.150 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.150 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.150 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.150 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.150 DEBUG:boto:Method: GET 2024-03-04T16:28:07.150 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.151 DEBUG:boto:Data: 2024-03-04T16:28:07.151 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.151 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.151 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.151 DEBUG:boto:Params: {} 2024-03-04T16:28:07.151 DEBUG:boto:Token: None 2024-03-04T16:28:07.152 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.152 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004995b8ebe2e98a4b-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.154 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.154 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.154 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.155 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.155 DEBUG:boto:Method: GET 2024-03-04T16:28:07.155 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.155 DEBUG:boto:Data: 2024-03-04T16:28:07.156 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.156 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.156 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.156 DEBUG:boto:Params: {} 2024-03-04T16:28:07.157 DEBUG:boto:Token: None 2024-03-04T16:28:07.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.157 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004456abbc1af02509-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.159 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.159 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.159 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.160 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.160 DEBUG:boto:Method: GET 2024-03-04T16:28:07.160 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.160 DEBUG:boto:Data: 2024-03-04T16:28:07.160 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.160 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.160 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.160 DEBUG:boto:Params: {} 2024-03-04T16:28:07.160 DEBUG:boto:Token: None 2024-03-04T16:28:07.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.161 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007134fad855af0872-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.164 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.164 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.164 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.164 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.165 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.165 DEBUG:boto:Method: GET 2024-03-04T16:28:07.165 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.165 DEBUG:boto:Data: 2024-03-04T16:28:07.165 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.165 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.165 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.165 DEBUG:boto:Params: {} 2024-03-04T16:28:07.165 DEBUG:boto:Token: None 2024-03-04T16:28:07.165 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.166 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a681d2e990772bfa-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.168 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:07.168 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.168 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.168 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.169 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.169 DEBUG:boto:Method: GET 2024-03-04T16:28:07.169 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.169 DEBUG:boto:Data: 2024-03-04T16:28:07.169 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.169 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.169 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.169 DEBUG:boto:Params: {} 2024-03-04T16:28:07.169 DEBUG:boto:Token: None 2024-03-04T16:28:07.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc4c234e04168097-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.172 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.172 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.172 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.172 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.172 DEBUG:boto:Method: GET 2024-03-04T16:28:07.172 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.173 DEBUG:boto:Data: 2024-03-04T16:28:07.173 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.173 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.173 DEBUG:boto:Params: {} 2024-03-04T16:28:07.173 DEBUG:boto:Token: None 2024-03-04T16:28:07.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4a5ce2e90733205-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.176 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.176 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.176 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.176 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.177 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.177 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.177 DEBUG:boto:Data: 2024-03-04T16:28:07.177 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.177 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.177 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.177 DEBUG:boto:Params: {} 2024-03-04T16:28:07.177 DEBUG:boto:Token: None 2024-03-04T16:28:07.178 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zWrrfGvWWkTcAF/UkPYo6GkwH2A= 2024-03-04T16:28:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zWrrfGvWWkTcAF/UkPYo6GkwH2A=', 'Content-Length': '0'} 2024-03-04T16:28:07.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec40afdbe4b42e10-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.180 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.180 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.180 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.180 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.181 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.181 DEBUG:boto:Data: 2024-03-04T16:28:07.181 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.181 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.181 DEBUG:boto:Params: {} 2024-03-04T16:28:07.181 DEBUG:boto:Token: None 2024-03-04T16:28:07.182 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zWrrfGvWWkTcAF/UkPYo6GkwH2A= 2024-03-04T16:28:07.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zWrrfGvWWkTcAF/UkPYo6GkwH2A=', 'Content-Length': '0'} 2024-03-04T16:28:07.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004726bf7be5311d6c-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:07.184 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.184 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.184 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.184 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.185 DEBUG:boto:Method: GET 2024-03-04T16:28:07.185 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.185 DEBUG:boto:Data: 2024-03-04T16:28:07.185 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.185 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.185 DEBUG:boto:Params: {} 2024-03-04T16:28:07.185 DEBUG:boto:Token: None 2024-03-04T16:28:07.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b653b90f8bf2b22-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.232 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.232 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.232 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.232 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.233 DEBUG:boto:Method: GET 2024-03-04T16:28:07.233 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.233 DEBUG:boto:Data: 2024-03-04T16:28:07.233 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.233 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.233 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.233 DEBUG:boto:Params: {} 2024-03-04T16:28:07.233 DEBUG:boto:Token: None 2024-03-04T16:28:07.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007d7a74f1ef41387-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.235 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.236 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.236 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.236 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.236 DEBUG:boto:Method: GET 2024-03-04T16:28:07.236 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.236 DEBUG:boto:Data: 2024-03-04T16:28:07.236 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.236 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.236 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.236 DEBUG:boto:Params: {} 2024-03-04T16:28:07.236 DEBUG:boto:Token: None 2024-03-04T16:28:07.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17399f8d3375e20-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.284 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.285 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.285 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.285 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.285 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.285 DEBUG:boto:Method: GET 2024-03-04T16:28:07.285 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.285 DEBUG:boto:Data: 2024-03-04T16:28:07.285 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.285 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.285 DEBUG:boto:Params: {} 2024-03-04T16:28:07.286 DEBUG:boto:Token: None 2024-03-04T16:28:07.286 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af5e833e8e7daca3-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.288 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.289 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.289 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.289 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.289 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.289 DEBUG:boto:Data: 2024-03-04T16:28:07.289 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.289 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.289 DEBUG:boto:Params: {} 2024-03-04T16:28:07.290 DEBUG:boto:Token: None 2024-03-04T16:28:07.290 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HCKTnIiubJja7GabtDzU5jqPn9Q= 2024-03-04T16:28:07.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HCKTnIiubJja7GabtDzU5jqPn9Q=', 'Content-Length': '0'} 2024-03-04T16:28:07.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057988966e068cd38-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.292 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.292 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.292 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.293 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.293 DEBUG:boto:Data: 2024-03-04T16:28:07.293 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.293 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.293 DEBUG:boto:Params: {} 2024-03-04T16:28:07.293 DEBUG:boto:Token: None 2024-03-04T16:28:07.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.293 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HCKTnIiubJja7GabtDzU5jqPn9Q= 2024-03-04T16:28:07.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HCKTnIiubJja7GabtDzU5jqPn9Q=', 'Content-Length': '0'} 2024-03-04T16:28:07.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073ebbdd2244913d9-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:07.296 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.296 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.296 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.296 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.296 DEBUG:boto:Method: GET 2024-03-04T16:28:07.296 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.296 DEBUG:boto:Data: 2024-03-04T16:28:07.297 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.297 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.297 DEBUG:boto:Params: {} 2024-03-04T16:28:07.297 DEBUG:boto:Token: None 2024-03-04T16:28:07.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000114559fafb17c6eb-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.340 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.340 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:07.340 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.340 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.341 DEBUG:boto:Method: GET 2024-03-04T16:28:07.341 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.341 DEBUG:boto:Data: 2024-03-04T16:28:07.341 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.341 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.341 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.341 DEBUG:boto:Params: {} 2024-03-04T16:28:07.341 DEBUG:boto:Token: None 2024-03-04T16:28:07.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:07.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw= 2024-03-04T16:28:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bKG1712DVCbW0fd8Rrd59+M8eFw=', 'Content-Length': '0'} 2024-03-04T16:28:07.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae8ce7e08f65a8a0-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.344 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.344 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.344 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.344 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.344 DEBUG:boto:Method: GET 2024-03-04T16:28:07.344 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.344 DEBUG:boto:Data: 2024-03-04T16:28:07.345 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.345 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.345 DEBUG:boto:Params: {} 2024-03-04T16:28:07.345 DEBUG:boto:Token: None 2024-03-04T16:28:07.345 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.346 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059766be275717955-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.348 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.348 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.349 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.349 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.349 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.349 DEBUG:boto:Method: GET 2024-03-04T16:28:07.349 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:07.349 DEBUG:boto:Data: 2024-03-04T16:28:07.350 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.350 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.350 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.350 DEBUG:boto:Params: {} 2024-03-04T16:28:07.350 DEBUG:boto:Token: None 2024-03-04T16:28:07.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a29bbfeeb5b03e-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.353 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:07.353 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.353 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.354 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.354 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.354 DEBUG:boto:Method: GET 2024-03-04T16:28:07.354 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.354 DEBUG:boto:Data: 2024-03-04T16:28:07.354 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.354 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.354 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.355 DEBUG:boto:Params: {} 2024-03-04T16:28:07.355 DEBUG:boto:Token: None 2024-03-04T16:28:07.355 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.355 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000012de93ad5a12dc-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.400 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.400 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.403 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.405 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.405 DEBUG:boto:Method: GET 2024-03-04T16:28:07.405 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.405 DEBUG:boto:Data: 2024-03-04T16:28:07.405 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.405 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.405 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.405 DEBUG:boto:Params: {} 2024-03-04T16:28:07.406 DEBUG:boto:Token: None 2024-03-04T16:28:07.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000971de10ff8d0a52c-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.452 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.452 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.452 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.452 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.453 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.453 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.453 DEBUG:boto:Data: 2024-03-04T16:28:07.453 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.453 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.453 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.453 DEBUG:boto:Params: {} 2024-03-04T16:28:07.453 DEBUG:boto:Token: None 2024-03-04T16:28:07.453 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zPV6XszgqC8r7xJ2EwblCWuWXMw= 2024-03-04T16:28:07.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zPV6XszgqC8r7xJ2EwblCWuWXMw=', 'Content-Length': '0'} 2024-03-04T16:28:07.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e50bcd813fbce04-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.456 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.456 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.456 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.456 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.456 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.456 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.456 DEBUG:boto:Data: 2024-03-04T16:28:07.456 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.456 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.457 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.457 DEBUG:boto:Params: {} 2024-03-04T16:28:07.457 DEBUG:boto:Token: None 2024-03-04T16:28:07.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zPV6XszgqC8r7xJ2EwblCWuWXMw= 2024-03-04T16:28:07.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zPV6XszgqC8r7xJ2EwblCWuWXMw=', 'Content-Length': '0'} 2024-03-04T16:28:07.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069954ee2ec88ab29-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:07.459 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.459 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.459 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.459 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.459 DEBUG:boto:Method: GET 2024-03-04T16:28:07.459 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.460 DEBUG:boto:Data: 2024-03-04T16:28:07.460 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.460 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.460 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.460 DEBUG:boto:Params: {} 2024-03-04T16:28:07.460 DEBUG:boto:Token: None 2024-03-04T16:28:07.460 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.460 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000527d863613f7c8d1-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.462 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.462 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.462 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.463 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.463 DEBUG:boto:Method: GET 2024-03-04T16:28:07.463 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.463 DEBUG:boto:Data: 2024-03-04T16:28:07.463 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.463 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.463 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.463 DEBUG:boto:Params: {} 2024-03-04T16:28:07.464 DEBUG:boto:Token: None 2024-03-04T16:28:07.464 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.464 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084f26a1d7a6218f0-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.512 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.512 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.512 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.512 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.513 DEBUG:boto:Method: GET 2024-03-04T16:28:07.513 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.513 DEBUG:boto:Data: 2024-03-04T16:28:07.513 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.513 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.513 DEBUG:boto:Params: {} 2024-03-04T16:28:07.513 DEBUG:boto:Token: None 2024-03-04T16:28:07.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ed808ad3705e80-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.516 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.516 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.516 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.517 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.517 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.517 DEBUG:boto:Method: GET 2024-03-04T16:28:07.517 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.517 DEBUG:boto:Data: 2024-03-04T16:28:07.517 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.517 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.517 DEBUG:boto:Params: {} 2024-03-04T16:28:07.517 DEBUG:boto:Token: None 2024-03-04T16:28:07.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ff2c13155098569-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.520 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.520 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.521 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.521 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.521 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.521 DEBUG:boto:Data: 2024-03-04T16:28:07.521 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.521 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.521 DEBUG:boto:Params: {} 2024-03-04T16:28:07.521 DEBUG:boto:Token: None 2024-03-04T16:28:07.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.522 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5k0em3yD3L5JsO4WDk/rqIJqwyE= 2024-03-04T16:28:07.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5k0em3yD3L5JsO4WDk/rqIJqwyE=', 'Content-Length': '0'} 2024-03-04T16:28:07.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000907d0255bc56ccf6-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.524 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.524 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.524 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.524 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.524 DEBUG:boto:Data: 2024-03-04T16:28:07.524 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.525 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.525 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.525 DEBUG:boto:Params: {} 2024-03-04T16:28:07.525 DEBUG:boto:Token: None 2024-03-04T16:28:07.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5k0em3yD3L5JsO4WDk/rqIJqwyE= 2024-03-04T16:28:07.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5k0em3yD3L5JsO4WDk/rqIJqwyE=', 'Content-Length': '0'} 2024-03-04T16:28:07.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005239c48b5448d60-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:07.528 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.528 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.528 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.528 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.528 DEBUG:boto:Method: GET 2024-03-04T16:28:07.529 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.529 DEBUG:boto:Data: 2024-03-04T16:28:07.529 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.529 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.529 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.530 DEBUG:boto:Params: {} 2024-03-04T16:28:07.530 DEBUG:boto:Token: None 2024-03-04T16:28:07.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da2e4677bf7336f9-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.532 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.533 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:07.533 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.533 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.533 DEBUG:boto:Method: GET 2024-03-04T16:28:07.533 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.533 DEBUG:boto:Data: 2024-03-04T16:28:07.533 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.534 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.534 DEBUG:boto:Params: {} 2024-03-04T16:28:07.534 DEBUG:boto:Token: None 2024-03-04T16:28:07.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:07.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI= 2024-03-04T16:28:07.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OxmYx9wBvbrC0uC/7Ym7kLOVjeI=', 'Content-Length': '0'} 2024-03-04T16:28:07.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b98c2fd65f1dccc-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.576 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.576 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.576 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.577 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.577 DEBUG:boto:Method: GET 2024-03-04T16:28:07.577 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.577 DEBUG:boto:Data: 2024-03-04T16:28:07.577 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.577 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.577 DEBUG:boto:Params: {} 2024-03-04T16:28:07.577 DEBUG:boto:Token: None 2024-03-04T16:28:07.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009528494c3b29ba27-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.580 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.580 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.580 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.581 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.581 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.581 DEBUG:boto:Method: GET 2024-03-04T16:28:07.581 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:07.581 DEBUG:boto:Data: 2024-03-04T16:28:07.581 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.581 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.581 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.581 DEBUG:boto:Params: {} 2024-03-04T16:28:07.581 DEBUG:boto:Token: None 2024-03-04T16:28:07.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.582 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b7d554454de7a36-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.585 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:07.585 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.585 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.586 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.586 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.586 DEBUG:boto:Method: GET 2024-03-04T16:28:07.586 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.586 DEBUG:boto:Data: 2024-03-04T16:28:07.586 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.586 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.586 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.586 DEBUG:boto:Params: {} 2024-03-04T16:28:07.586 DEBUG:boto:Token: None 2024-03-04T16:28:07.587 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.587 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ebb243425a0bc71-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.589 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.590 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.590 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.590 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.590 DEBUG:boto:Method: GET 2024-03-04T16:28:07.590 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.590 DEBUG:boto:Data: 2024-03-04T16:28:07.590 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.590 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.590 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.590 DEBUG:boto:Params: {} 2024-03-04T16:28:07.591 DEBUG:boto:Token: None 2024-03-04T16:28:07.591 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.591 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024a86a60c71eca4a-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.636 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.636 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.636 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.636 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.637 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.637 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.637 DEBUG:boto:Data: 2024-03-04T16:28:07.637 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.637 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.637 DEBUG:boto:Params: {} 2024-03-04T16:28:07.637 DEBUG:boto:Token: None 2024-03-04T16:28:07.637 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C+P8G5Qr/de+SFngZTL/2F+nFGM= 2024-03-04T16:28:07.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C+P8G5Qr/de+SFngZTL/2F+nFGM=', 'Content-Length': '0'} 2024-03-04T16:28:07.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc6b16914384cf9c-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.639 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.640 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.640 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.640 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.640 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.641 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.641 DEBUG:boto:Data: 2024-03-04T16:28:07.641 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.641 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.641 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.641 DEBUG:boto:Params: {} 2024-03-04T16:28:07.641 DEBUG:boto:Token: None 2024-03-04T16:28:07.642 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C+P8G5Qr/de+SFngZTL/2F+nFGM= 2024-03-04T16:28:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C+P8G5Qr/de+SFngZTL/2F+nFGM=', 'Content-Length': '0'} 2024-03-04T16:28:07.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb807c8392e9b09-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:07.645 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.645 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.645 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.645 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.645 DEBUG:boto:Method: GET 2024-03-04T16:28:07.645 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.645 DEBUG:boto:Data: 2024-03-04T16:28:07.645 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.645 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.646 DEBUG:boto:Params: {} 2024-03-04T16:28:07.646 DEBUG:boto:Token: None 2024-03-04T16:28:07.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fe23491784c1619-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.689 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.689 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.689 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.689 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.689 DEBUG:boto:Method: GET 2024-03-04T16:28:07.689 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.689 DEBUG:boto:Data: 2024-03-04T16:28:07.689 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.690 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.690 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.690 DEBUG:boto:Params: {} 2024-03-04T16:28:07.690 DEBUG:boto:Token: None 2024-03-04T16:28:07.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb12dde5bb03a390-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.740 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.740 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.740 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.740 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.741 DEBUG:boto:Method: GET 2024-03-04T16:28:07.741 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.741 DEBUG:boto:Data: 2024-03-04T16:28:07.741 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.741 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.741 DEBUG:boto:Params: {} 2024-03-04T16:28:07.741 DEBUG:boto:Token: None 2024-03-04T16:28:07.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.741 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000c66ddec394e042-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.744 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.745 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.745 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.745 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.745 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.745 DEBUG:boto:Method: GET 2024-03-04T16:28:07.745 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.745 DEBUG:boto:Data: 2024-03-04T16:28:07.745 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.745 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.745 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.745 DEBUG:boto:Params: {} 2024-03-04T16:28:07.746 DEBUG:boto:Token: None 2024-03-04T16:28:07.747 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.747 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d20b71a2929e9e0-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.749 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.750 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.750 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.750 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.750 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.750 DEBUG:boto:Data: 2024-03-04T16:28:07.750 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.750 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.751 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.751 DEBUG:boto:Params: {} 2024-03-04T16:28:07.751 DEBUG:boto:Token: None 2024-03-04T16:28:07.751 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.751 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sBnMFQvhD+dTvv68F/BVzCiyDnU= 2024-03-04T16:28:07.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sBnMFQvhD+dTvv68F/BVzCiyDnU=', 'Content-Length': '0'} 2024-03-04T16:28:07.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c314914bea5f22c-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.753 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.754 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.754 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.754 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.754 DEBUG:boto:Data: 2024-03-04T16:28:07.754 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.754 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.754 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.754 DEBUG:boto:Params: {} 2024-03-04T16:28:07.754 DEBUG:boto:Token: None 2024-03-04T16:28:07.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sBnMFQvhD+dTvv68F/BVzCiyDnU= 2024-03-04T16:28:07.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sBnMFQvhD+dTvv68F/BVzCiyDnU=', 'Content-Length': '0'} 2024-03-04T16:28:07.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004488f675e5b6ab5b-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:07.758 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.758 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.758 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.758 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.758 DEBUG:boto:Method: GET 2024-03-04T16:28:07.759 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.759 DEBUG:boto:Data: 2024-03-04T16:28:07.759 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.759 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.759 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.759 DEBUG:boto:Params: {} 2024-03-04T16:28:07.759 DEBUG:boto:Token: None 2024-03-04T16:28:07.759 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.760 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9c6d98ef4d2e2ad-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.804 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.804 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:07.804 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.805 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.805 DEBUG:boto:Method: GET 2024-03-04T16:28:07.805 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.805 DEBUG:boto:Data: 2024-03-04T16:28:07.805 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.805 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.805 DEBUG:boto:Params: {} 2024-03-04T16:28:07.805 DEBUG:boto:Token: None 2024-03-04T16:28:07.806 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:07.806 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A= 2024-03-04T16:28:07.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xktyx1SGbEgkJOm8o0QvYVi1I7A=', 'Content-Length': '0'} 2024-03-04T16:28:07.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a9d7d4748a4374e-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.848 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.848 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.848 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.848 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.849 DEBUG:boto:Method: GET 2024-03-04T16:28:07.849 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.849 DEBUG:boto:Data: 2024-03-04T16:28:07.849 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.849 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:07.849 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.849 DEBUG:boto:Params: {} 2024-03-04T16:28:07.850 DEBUG:boto:Token: None 2024-03-04T16:28:07.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad76d2abf2455ad-0065e5f697-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.896 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.897 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.897 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.897 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.897 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.897 DEBUG:boto:Method: GET 2024-03-04T16:28:07.897 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:07.897 DEBUG:boto:Data: 2024-03-04T16:28:07.897 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.897 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.897 DEBUG:boto:Params: {} 2024-03-04T16:28:07.898 DEBUG:boto:Token: None 2024-03-04T16:28:07.898 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dfb8e9444af4ff5-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.901 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone2, test-zone1} 2024-03-04T16:28:07.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone3, test-zone1} 2024-03-04T16:28:07.902 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.902 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.902 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.902 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:07.902 DEBUG:boto:Data: 2024-03-04T16:28:07.902 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.902 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.902 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.902 DEBUG:boto:Params: {} 2024-03-04T16:28:07.903 DEBUG:boto:Token: None 2024-03-04T16:28:07.903 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.903 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4= 2024-03-04T16:28:07.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4=', 'Content-Length': '0'} 2024-03-04T16:28:07.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001e54963e0d564da-0065e5f697-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.906 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.906 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.906 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.906 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:07.906 DEBUG:boto:Data: 2024-03-04T16:28:07.906 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.907 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.907 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.907 DEBUG:boto:Params: {} 2024-03-04T16:28:07.907 DEBUG:boto:Token: None 2024-03-04T16:28:07.907 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.907 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4= 2024-03-04T16:28:07.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jURUN63KtCYiC8cquDcSejAUIO4=', 'Content-Length': '0'} 2024-03-04T16:28:07.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b7179bec01aae7b-0065e5f697-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:07.910 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.910 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.910 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.910 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.910 DEBUG:boto:Method: GET 2024-03-04T16:28:07.910 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.911 DEBUG:boto:Data: 2024-03-04T16:28:07.911 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.911 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.911 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.911 DEBUG:boto:Params: {} 2024-03-04T16:28:07.911 DEBUG:boto:Token: None 2024-03-04T16:28:07.911 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.911 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab6253a3db3bd664-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.914 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.914 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:07.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:07.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:07.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:07.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:07.915 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.915 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.915 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.915 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.915 DEBUG:boto:Method: GET 2024-03-04T16:28:07.915 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.915 DEBUG:boto:Data: 2024-03-04T16:28:07.916 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.916 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.916 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.916 DEBUG:boto:Params: {} 2024-03-04T16:28:07.916 DEBUG:boto:Token: None 2024-03-04T16:28:07.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.916 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da1660d67e23182-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.919 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.919 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:07.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:07.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:07.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:07.920 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.920 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.920 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.920 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.920 DEBUG:boto:Method: GET 2024-03-04T16:28:07.920 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.920 DEBUG:boto:Data: 2024-03-04T16:28:07.920 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.921 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.921 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.921 DEBUG:boto:Params: {} 2024-03-04T16:28:07.921 DEBUG:boto:Token: None 2024-03-04T16:28:07.921 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8db37c77cebf08f-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.924 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.924 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.924 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.925 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.925 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.925 DEBUG:boto:Method: GET 2024-03-04T16:28:07.925 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:07.925 DEBUG:boto:Data: 2024-03-04T16:28:07.925 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.925 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.925 DEBUG:boto:Params: {} 2024-03-04T16:28:07.925 DEBUG:boto:Token: None 2024-03-04T16:28:07.926 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/?versions 2024-03-04T16:28:07.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0= 2024-03-04T16:28:07.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+uql/7GFpjVkejE/NYzGGowc7D0=', 'Content-Length': '0'} 2024-03-04T16:28:07.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077fa1abfaa0090f-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.929 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:07.929 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.929 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.929 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.930 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.930 DEBUG:boto:Method: GET 2024-03-04T16:28:07.930 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.930 DEBUG:boto:Data: 2024-03-04T16:28:07.930 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.930 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.930 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.930 DEBUG:boto:Params: {} 2024-03-04T16:28:07.930 DEBUG:boto:Token: None 2024-03-04T16:28:07.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.931 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d148a7e8acf759a-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.933 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.933 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.933 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.933 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.933 DEBUG:boto:Method: GET 2024-03-04T16:28:07.933 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.933 DEBUG:boto:Data: 2024-03-04T16:28:07.933 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.933 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.933 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.933 DEBUG:boto:Params: {} 2024-03-04T16:28:07.934 DEBUG:boto:Token: None 2024-03-04T16:28:07.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c15d20d88cc6154-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.936 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.936 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.936 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.937 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.937 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.937 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.937 DEBUG:boto:Data: 2024-03-04T16:28:07.937 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.937 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.937 DEBUG:boto:Params: {} 2024-03-04T16:28:07.937 DEBUG:boto:Token: None 2024-03-04T16:28:07.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g= 2024-03-04T16:28:07.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g=', 'Content-Length': '0'} 2024-03-04T16:28:07.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098608762f1313375-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.940 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.940 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.940 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.940 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.940 DEBUG:boto:Method: HEAD 2024-03-04T16:28:07.941 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.941 DEBUG:boto:Data: 2024-03-04T16:28:07.941 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.941 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.941 DEBUG:boto:Params: {} 2024-03-04T16:28:07.941 DEBUG:boto:Token: None 2024-03-04T16:28:07.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g= 2024-03-04T16:28:07.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rd4WDSYWwot/CHKxDw8TEDdvs6g=', 'Content-Length': '0'} 2024-03-04T16:28:07.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000217e45f54c4e441-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:07.943 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.944 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.944 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.944 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.944 DEBUG:boto:Method: GET 2024-03-04T16:28:07.944 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.944 DEBUG:boto:Data: 2024-03-04T16:28:07.944 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.944 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.944 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.944 DEBUG:boto:Params: {} 2024-03-04T16:28:07.945 DEBUG:boto:Token: None 2024-03-04T16:28:07.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003aa99a2f3eed9fb3-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.947 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:07.947 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:07.948 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.948 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.948 DEBUG:boto:Method: GET 2024-03-04T16:28:07.948 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.948 DEBUG:boto:Data: 2024-03-04T16:28:07.948 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.948 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.949 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.949 DEBUG:boto:Params: {} 2024-03-04T16:28:07.949 DEBUG:boto:Token: None 2024-03-04T16:28:07.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:07.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE= 2024-03-04T16:28:07.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8GUVYSQC26b9wP/coZ3O0IqONE=', 'Content-Length': '0'} 2024-03-04T16:28:07.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b81991f44d3cca8-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.951 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.951 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.951 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.952 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.952 DEBUG:boto:Method: GET 2024-03-04T16:28:07.952 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.952 DEBUG:boto:Data: 2024-03-04T16:28:07.952 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.952 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:07.953 DEBUG:boto:Port: 8001 2024-03-04T16:28:07.953 DEBUG:boto:Params: {} 2024-03-04T16:28:07.953 DEBUG:boto:Token: None 2024-03-04T16:28:07.953 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.953 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037e911cb6d7b03f8-0065e5f697-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:07.957 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:07.958 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:07.958 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:07.958 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.958 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.958 DEBUG:boto:Method: GET 2024-03-04T16:28:07.958 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:07.958 DEBUG:boto:Data: 2024-03-04T16:28:07.959 DEBUG:boto:Headers: {} 2024-03-04T16:28:07.959 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:07.959 DEBUG:boto:Port: 8000 2024-03-04T16:28:07.959 DEBUG:boto:Params: {} 2024-03-04T16:28:07.959 DEBUG:boto:Token: None 2024-03-04T16:28:07.959 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:07 GMT /sbnvju-20/ 2024-03-04T16:28:07.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs= 2024-03-04T16:28:07.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:07 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ln2tb99VYP+Zjxtx/78sGG6SuDs=', 'Content-Length': '0'} 2024-03-04T16:28:07.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030f52a5d1b8c8b65-0065e5f697-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:07 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.008 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.009 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:08.009 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:08.009 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.009 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:08.009 DEBUG:boto:Data: 2024-03-04T16:28:08.009 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.009 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.010 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.010 DEBUG:boto:Params: {} 2024-03-04T16:28:08.010 DEBUG:boto:Token: None 2024-03-04T16:28:08.010 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/dummy 2024-03-04T16:28:08.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1U4wh/zMXQ6qDWJheRMb2nQC82Y= 2024-03-04T16:28:08.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1U4wh/zMXQ6qDWJheRMb2nQC82Y=', 'Content-Length': '0'} 2024-03-04T16:28:08.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000698c826a695be6cb-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.016 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:08.016 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:08.016 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.016 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:08.016 DEBUG:boto:Data: 2024-03-04T16:28:08.016 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.016 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.016 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.017 DEBUG:boto:Params: {} 2024-03-04T16:28:08.017 DEBUG:boto:Token: None 2024-03-04T16:28:08.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/dummy 2024-03-04T16:28:08.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1U4wh/zMXQ6qDWJheRMb2nQC82Y= 2024-03-04T16:28:08.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1U4wh/zMXQ6qDWJheRMb2nQC82Y=', 'Content-Length': '0'} 2024-03-04T16:28:08.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3e72d6ccd14ae3b-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:08.020 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:08.020 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:08.020 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.020 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.020 DEBUG:boto:Method: GET 2024-03-04T16:28:08.020 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.021 DEBUG:boto:Data: 2024-03-04T16:28:08.021 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.021 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.021 DEBUG:boto:Params: {} 2024-03-04T16:28:08.021 DEBUG:boto:Token: None 2024-03-04T16:28:08.021 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.021 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s= 2024-03-04T16:28:08.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s=', 'Content-Length': '0'} 2024-03-04T16:28:08.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6cf4317c7412395-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.023 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:08.024 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:08.024 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.024 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.024 DEBUG:boto:Method: GET 2024-03-04T16:28:08.024 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.024 DEBUG:boto:Data: 2024-03-04T16:28:08.024 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.024 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.024 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.024 DEBUG:boto:Params: {} 2024-03-04T16:28:08.024 DEBUG:boto:Token: None 2024-03-04T16:28:08.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s= 2024-03-04T16:28:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s=', 'Content-Length': '0'} 2024-03-04T16:28:08.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9638ec408d85c01-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.027 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.027 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.027 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.027 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.028 DEBUG:boto:Method: GET 2024-03-04T16:28:08.028 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.028 DEBUG:boto:Data: 2024-03-04T16:28:08.028 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.028 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.028 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.028 DEBUG:boto:Params: {} 2024-03-04T16:28:08.028 DEBUG:boto:Token: None 2024-03-04T16:28:08.028 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.028 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde3486eb1c793a5-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.072 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.073 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.073 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.073 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.073 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.074 DEBUG:boto:Method: GET 2024-03-04T16:28:08.074 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:08.074 DEBUG:boto:Data: 2024-03-04T16:28:08.074 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.074 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.074 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.074 DEBUG:boto:Params: {} 2024-03-04T16:28:08.074 DEBUG:boto:Token: None 2024-03-04T16:28:08.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67712752ce03ed1-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.077 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:08.078 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.078 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.078 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.078 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.078 DEBUG:boto:Method: GET 2024-03-04T16:28:08.079 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.079 DEBUG:boto:Data: 2024-03-04T16:28:08.079 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.079 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.079 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.079 DEBUG:boto:Params: {} 2024-03-04T16:28:08.079 DEBUG:boto:Token: None 2024-03-04T16:28:08.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d995484c49297b49-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.128 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.128 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.128 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.128 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.129 DEBUG:boto:Method: GET 2024-03-04T16:28:08.129 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.129 DEBUG:boto:Data: 2024-03-04T16:28:08.129 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.129 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.129 DEBUG:boto:Params: {} 2024-03-04T16:28:08.129 DEBUG:boto:Token: None 2024-03-04T16:28:08.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009289d1dce23d22bc-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.132 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.132 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.132 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.132 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.132 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.132 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.132 DEBUG:boto:Data: 2024-03-04T16:28:08.132 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.132 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.133 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.133 DEBUG:boto:Params: {} 2024-03-04T16:28:08.133 DEBUG:boto:Token: None 2024-03-04T16:28:08.133 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HMg7rT2O9l3FtHfA298r9od1kws= 2024-03-04T16:28:08.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HMg7rT2O9l3FtHfA298r9od1kws=', 'Content-Length': '0'} 2024-03-04T16:28:08.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cdc27cbc002ad49-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.135 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.135 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.136 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.136 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.136 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.136 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.136 DEBUG:boto:Data: 2024-03-04T16:28:08.136 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.136 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.136 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.136 DEBUG:boto:Params: {} 2024-03-04T16:28:08.136 DEBUG:boto:Token: None 2024-03-04T16:28:08.137 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HMg7rT2O9l3FtHfA298r9od1kws= 2024-03-04T16:28:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HMg7rT2O9l3FtHfA298r9od1kws=', 'Content-Length': '0'} 2024-03-04T16:28:08.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087da94559f18f81b-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:08.139 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.139 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.139 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.139 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.139 DEBUG:boto:Method: GET 2024-03-04T16:28:08.140 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.140 DEBUG:boto:Data: 2024-03-04T16:28:08.140 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.140 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.140 DEBUG:boto:Params: {} 2024-03-04T16:28:08.140 DEBUG:boto:Token: None 2024-03-04T16:28:08.140 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.140 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e51a3c9ca6b8034-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.184 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.184 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.184 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.185 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.185 DEBUG:boto:Method: GET 2024-03-04T16:28:08.185 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.185 DEBUG:boto:Data: 2024-03-04T16:28:08.185 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.185 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.185 DEBUG:boto:Params: {} 2024-03-04T16:28:08.185 DEBUG:boto:Token: None 2024-03-04T16:28:08.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002eb0f2a28b4ffda9-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.188 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.188 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.188 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.188 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.188 DEBUG:boto:Method: GET 2024-03-04T16:28:08.188 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.188 DEBUG:boto:Data: 2024-03-04T16:28:08.189 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.189 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.192 DEBUG:boto:Params: {} 2024-03-04T16:28:08.192 DEBUG:boto:Token: None 2024-03-04T16:28:08.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.193 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc6d59126c33ca4-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.236 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.237 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.237 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.237 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.237 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.237 DEBUG:boto:Method: GET 2024-03-04T16:28:08.237 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.237 DEBUG:boto:Data: 2024-03-04T16:28:08.238 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.238 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.238 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.238 DEBUG:boto:Params: {} 2024-03-04T16:28:08.238 DEBUG:boto:Token: None 2024-03-04T16:28:08.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.239 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5912e14c68ed880-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.243 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.243 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.244 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.244 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.244 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.244 DEBUG:boto:Data: 2024-03-04T16:28:08.244 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.244 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.244 DEBUG:boto:Params: {} 2024-03-04T16:28:08.245 DEBUG:boto:Token: None 2024-03-04T16:28:08.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nYEuJEOast8C+xlkWwbTR9zZuWk= 2024-03-04T16:28:08.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nYEuJEOast8C+xlkWwbTR9zZuWk=', 'Content-Length': '0'} 2024-03-04T16:28:08.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040a1b05bebfd15d6-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.248 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.248 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.248 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.248 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.248 DEBUG:boto:Data: 2024-03-04T16:28:08.248 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.249 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.249 DEBUG:boto:Params: {} 2024-03-04T16:28:08.249 DEBUG:boto:Token: None 2024-03-04T16:28:08.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nYEuJEOast8C+xlkWwbTR9zZuWk= 2024-03-04T16:28:08.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nYEuJEOast8C+xlkWwbTR9zZuWk=', 'Content-Length': '0'} 2024-03-04T16:28:08.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddc2071363b87a17-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:08.252 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.252 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.252 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.253 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.253 DEBUG:boto:Method: GET 2024-03-04T16:28:08.253 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.253 DEBUG:boto:Data: 2024-03-04T16:28:08.253 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.253 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.253 DEBUG:boto:Params: {} 2024-03-04T16:28:08.253 DEBUG:boto:Token: None 2024-03-04T16:28:08.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2631ca72853779f-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.300 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.300 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:08.300 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.300 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.301 DEBUG:boto:Method: GET 2024-03-04T16:28:08.301 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.301 DEBUG:boto:Data: 2024-03-04T16:28:08.301 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.301 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.301 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.301 DEBUG:boto:Params: {} 2024-03-04T16:28:08.301 DEBUG:boto:Token: None 2024-03-04T16:28:08.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:08.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ= 2024-03-04T16:28:08.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ztd0UBINP2usugaON/wvf8HhlqQ=', 'Content-Length': '0'} 2024-03-04T16:28:08.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008961fe1bdb5e208e-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.304 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.304 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.304 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.304 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.304 DEBUG:boto:Method: GET 2024-03-04T16:28:08.305 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.305 DEBUG:boto:Data: 2024-03-04T16:28:08.305 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.305 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.305 DEBUG:boto:Params: {} 2024-03-04T16:28:08.305 DEBUG:boto:Token: None 2024-03-04T16:28:08.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.305 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b1df9ce8dac505-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.308 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.309 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.309 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.309 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.309 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.309 DEBUG:boto:Method: GET 2024-03-04T16:28:08.309 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:08.309 DEBUG:boto:Data: 2024-03-04T16:28:08.309 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.310 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.310 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.310 DEBUG:boto:Params: {} 2024-03-04T16:28:08.310 DEBUG:boto:Token: None 2024-03-04T16:28:08.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d80d55214685844-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.313 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:08.313 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.313 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.313 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.313 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.313 DEBUG:boto:Method: GET 2024-03-04T16:28:08.314 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.314 DEBUG:boto:Data: 2024-03-04T16:28:08.314 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.314 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.314 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.314 DEBUG:boto:Params: {} 2024-03-04T16:28:08.314 DEBUG:boto:Token: None 2024-03-04T16:28:08.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000176f1bd9215e7169-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.360 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.360 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.361 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.361 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.361 DEBUG:boto:Method: GET 2024-03-04T16:28:08.361 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.361 DEBUG:boto:Data: 2024-03-04T16:28:08.361 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.361 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.361 DEBUG:boto:Params: {} 2024-03-04T16:28:08.362 DEBUG:boto:Token: None 2024-03-04T16:28:08.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b137fc52e06118c-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.409 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.409 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.409 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.409 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.409 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.409 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.410 DEBUG:boto:Data: 2024-03-04T16:28:08.410 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.410 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.410 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.410 DEBUG:boto:Params: {} 2024-03-04T16:28:08.410 DEBUG:boto:Token: None 2024-03-04T16:28:08.410 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.411 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Su0w0nOBZ/ivG+UlRWl+P1WxYoU= 2024-03-04T16:28:08.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Su0w0nOBZ/ivG+UlRWl+P1WxYoU=', 'Content-Length': '0'} 2024-03-04T16:28:08.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ad3d973e984fdc3-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.413 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.413 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.414 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.414 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.414 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.414 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.414 DEBUG:boto:Data: 2024-03-04T16:28:08.414 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.414 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.414 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.414 DEBUG:boto:Params: {} 2024-03-04T16:28:08.415 DEBUG:boto:Token: None 2024-03-04T16:28:08.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Su0w0nOBZ/ivG+UlRWl+P1WxYoU= 2024-03-04T16:28:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Su0w0nOBZ/ivG+UlRWl+P1WxYoU=', 'Content-Length': '0'} 2024-03-04T16:28:08.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025404a5eaaa48452-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:08.418 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.418 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.418 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.418 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.418 DEBUG:boto:Method: GET 2024-03-04T16:28:08.418 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.418 DEBUG:boto:Data: 2024-03-04T16:28:08.418 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.418 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.418 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.418 DEBUG:boto:Params: {} 2024-03-04T16:28:08.419 DEBUG:boto:Token: None 2024-03-04T16:28:08.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c129ac8a295f78c-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.464 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.464 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.464 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.464 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.465 DEBUG:boto:Method: GET 2024-03-04T16:28:08.465 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.465 DEBUG:boto:Data: 2024-03-04T16:28:08.465 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.465 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.465 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.465 DEBUG:boto:Params: {} 2024-03-04T16:28:08.465 DEBUG:boto:Token: None 2024-03-04T16:28:08.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aa018805ed19613-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.508 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.508 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.508 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.508 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.509 DEBUG:boto:Method: GET 2024-03-04T16:28:08.509 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.509 DEBUG:boto:Data: 2024-03-04T16:28:08.509 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.509 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.509 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.509 DEBUG:boto:Params: {} 2024-03-04T16:28:08.509 DEBUG:boto:Token: None 2024-03-04T16:28:08.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479d7d294a9378e7-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.513 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.514 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.514 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.514 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.514 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.514 DEBUG:boto:Method: GET 2024-03-04T16:28:08.514 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.514 DEBUG:boto:Data: 2024-03-04T16:28:08.514 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.515 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.515 DEBUG:boto:Params: {} 2024-03-04T16:28:08.515 DEBUG:boto:Token: None 2024-03-04T16:28:08.515 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce6f99a4c7ef7ed-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.560 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.560 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.561 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.561 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.561 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.561 DEBUG:boto:Data: 2024-03-04T16:28:08.561 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.561 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.561 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.561 DEBUG:boto:Params: {} 2024-03-04T16:28:08.562 DEBUG:boto:Token: None 2024-03-04T16:28:08.562 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TORNHUDFENVf8Z+k1JrfIGO8iro= 2024-03-04T16:28:08.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TORNHUDFENVf8Z+k1JrfIGO8iro=', 'Content-Length': '0'} 2024-03-04T16:28:08.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007ffc4eab1b33523-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.566 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.566 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.566 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.566 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.566 DEBUG:boto:Data: 2024-03-04T16:28:08.566 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.566 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.567 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.567 DEBUG:boto:Params: {} 2024-03-04T16:28:08.567 DEBUG:boto:Token: None 2024-03-04T16:28:08.567 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TORNHUDFENVf8Z+k1JrfIGO8iro= 2024-03-04T16:28:08.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TORNHUDFENVf8Z+k1JrfIGO8iro=', 'Content-Length': '0'} 2024-03-04T16:28:08.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000539e1c65943d8234-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:08.573 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.574 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.574 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.574 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.574 DEBUG:boto:Method: GET 2024-03-04T16:28:08.574 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.574 DEBUG:boto:Data: 2024-03-04T16:28:08.574 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.574 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.574 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.574 DEBUG:boto:Params: {} 2024-03-04T16:28:08.574 DEBUG:boto:Token: None 2024-03-04T16:28:08.575 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.577 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065e6b7430479f322-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.620 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.621 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:08.621 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.621 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.621 DEBUG:boto:Method: GET 2024-03-04T16:28:08.621 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.621 DEBUG:boto:Data: 2024-03-04T16:28:08.621 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.621 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.621 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.621 DEBUG:boto:Params: {} 2024-03-04T16:28:08.621 DEBUG:boto:Token: None 2024-03-04T16:28:08.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:08.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY= 2024-03-04T16:28:08.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZtV8bP4eGjbGAbmmyL0LvUYm9OY=', 'Content-Length': '0'} 2024-03-04T16:28:08.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000346d4f7517428fe4-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.668 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.668 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.669 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.669 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.669 DEBUG:boto:Method: GET 2024-03-04T16:28:08.669 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.669 DEBUG:boto:Data: 2024-03-04T16:28:08.669 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.669 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.669 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.669 DEBUG:boto:Params: {} 2024-03-04T16:28:08.669 DEBUG:boto:Token: None 2024-03-04T16:28:08.670 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.670 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d712054163e97687-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.673 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.673 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.673 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.673 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.674 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.674 DEBUG:boto:Method: GET 2024-03-04T16:28:08.674 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:08.674 DEBUG:boto:Data: 2024-03-04T16:28:08.674 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.674 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.674 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.674 DEBUG:boto:Params: {} 2024-03-04T16:28:08.674 DEBUG:boto:Token: None 2024-03-04T16:28:08.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.675 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46dea4866ab98b8-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.720 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:08.721 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.721 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.721 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.721 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.721 DEBUG:boto:Method: GET 2024-03-04T16:28:08.721 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.721 DEBUG:boto:Data: 2024-03-04T16:28:08.721 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.722 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.722 DEBUG:boto:Params: {} 2024-03-04T16:28:08.722 DEBUG:boto:Token: None 2024-03-04T16:28:08.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed0d6093de4d6c12-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.725 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.725 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.725 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.725 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.725 DEBUG:boto:Method: GET 2024-03-04T16:28:08.725 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.725 DEBUG:boto:Data: 2024-03-04T16:28:08.725 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.725 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.726 DEBUG:boto:Params: {} 2024-03-04T16:28:08.726 DEBUG:boto:Token: None 2024-03-04T16:28:08.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000006489438f4d7a10-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.772 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.772 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.773 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.773 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.774 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.774 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.774 DEBUG:boto:Data: 2024-03-04T16:28:08.775 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.775 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.775 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.775 DEBUG:boto:Params: {} 2024-03-04T16:28:08.776 DEBUG:boto:Token: None 2024-03-04T16:28:08.776 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.776 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M/C8N0PC7/eYFREgqbuSSJqQWiE= 2024-03-04T16:28:08.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M/C8N0PC7/eYFREgqbuSSJqQWiE=', 'Content-Length': '0'} 2024-03-04T16:28:08.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc617bd4768eeb0e-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.779 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.779 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.779 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.780 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.780 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.780 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.780 DEBUG:boto:Data: 2024-03-04T16:28:08.781 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.781 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.781 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.781 DEBUG:boto:Params: {} 2024-03-04T16:28:08.781 DEBUG:boto:Token: None 2024-03-04T16:28:08.782 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M/C8N0PC7/eYFREgqbuSSJqQWiE= 2024-03-04T16:28:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M/C8N0PC7/eYFREgqbuSSJqQWiE=', 'Content-Length': '0'} 2024-03-04T16:28:08.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000424e5835c1ca2a92-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:08.785 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.785 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.785 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.785 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.785 DEBUG:boto:Method: GET 2024-03-04T16:28:08.785 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.785 DEBUG:boto:Data: 2024-03-04T16:28:08.785 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.786 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.786 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.786 DEBUG:boto:Params: {} 2024-03-04T16:28:08.786 DEBUG:boto:Token: None 2024-03-04T16:28:08.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2d9e1b8117f1b6b-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.789 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.789 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.789 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.789 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.789 DEBUG:boto:Method: GET 2024-03-04T16:28:08.789 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.789 DEBUG:boto:Data: 2024-03-04T16:28:08.789 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.790 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.790 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.790 DEBUG:boto:Params: {} 2024-03-04T16:28:08.790 DEBUG:boto:Token: None 2024-03-04T16:28:08.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f74e91e0b17773d-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.840 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.840 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.840 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.841 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.841 DEBUG:boto:Method: GET 2024-03-04T16:28:08.841 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.841 DEBUG:boto:Data: 2024-03-04T16:28:08.841 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.841 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.841 DEBUG:boto:Params: {} 2024-03-04T16:28:08.842 DEBUG:boto:Token: None 2024-03-04T16:28:08.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004df99dbd45e0a635-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.845 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.846 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.846 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.846 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.846 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.846 DEBUG:boto:Method: GET 2024-03-04T16:28:08.846 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.846 DEBUG:boto:Data: 2024-03-04T16:28:08.846 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.846 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.847 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.847 DEBUG:boto:Params: {} 2024-03-04T16:28:08.847 DEBUG:boto:Token: None 2024-03-04T16:28:08.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091aa2f24b153af01-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.850 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.851 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.851 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.851 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.851 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.851 DEBUG:boto:Data: 2024-03-04T16:28:08.851 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.851 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.852 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.852 DEBUG:boto:Params: {} 2024-03-04T16:28:08.852 DEBUG:boto:Token: None 2024-03-04T16:28:08.852 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.852 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lnZo9TpNjjBXj+Gb7lmMseXRXzo= 2024-03-04T16:28:08.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lnZo9TpNjjBXj+Gb7lmMseXRXzo=', 'Content-Length': '0'} 2024-03-04T16:28:08.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc03076765adef0d-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.855 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.855 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.855 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.856 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.856 DEBUG:boto:Data: 2024-03-04T16:28:08.856 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.856 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.856 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.856 DEBUG:boto:Params: {} 2024-03-04T16:28:08.856 DEBUG:boto:Token: None 2024-03-04T16:28:08.857 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lnZo9TpNjjBXj+Gb7lmMseXRXzo= 2024-03-04T16:28:08.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lnZo9TpNjjBXj+Gb7lmMseXRXzo=', 'Content-Length': '0'} 2024-03-04T16:28:08.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097df779b40cf8cc6-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:08.860 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.860 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.861 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.861 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.861 DEBUG:boto:Method: GET 2024-03-04T16:28:08.861 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.861 DEBUG:boto:Data: 2024-03-04T16:28:08.861 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.861 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.861 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.862 DEBUG:boto:Params: {} 2024-03-04T16:28:08.862 DEBUG:boto:Token: None 2024-03-04T16:28:08.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008047bda24ac905f3-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.864 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.864 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:08.864 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.864 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.865 DEBUG:boto:Method: GET 2024-03-04T16:28:08.865 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.865 DEBUG:boto:Data: 2024-03-04T16:28:08.865 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.865 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.865 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.865 DEBUG:boto:Params: {} 2024-03-04T16:28:08.865 DEBUG:boto:Token: None 2024-03-04T16:28:08.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:08.866 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U= 2024-03-04T16:28:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2PcB1NDv8q0mPaYu4nO2OlA+/7U=', 'Content-Length': '0'} 2024-03-04T16:28:08.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c4f4e0ee3deee15-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.912 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.912 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.912 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.913 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.913 DEBUG:boto:Method: GET 2024-03-04T16:28:08.913 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.913 DEBUG:boto:Data: 2024-03-04T16:28:08.913 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.913 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.913 DEBUG:boto:Params: {} 2024-03-04T16:28:08.913 DEBUG:boto:Token: None 2024-03-04T16:28:08.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd50ede863277a7-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.916 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.917 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.917 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.917 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.917 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.917 DEBUG:boto:Method: GET 2024-03-04T16:28:08.917 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:08.917 DEBUG:boto:Data: 2024-03-04T16:28:08.918 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.918 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:08.918 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.918 DEBUG:boto:Params: {} 2024-03-04T16:28:08.918 DEBUG:boto:Token: None 2024-03-04T16:28:08.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.918 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY= 2024-03-04T16:28:08.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uRhUwpgxAJybOznQJsZ4VvxR1dY=', 'Content-Length': '0'} 2024-03-04T16:28:08.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3bb24a818108f6-0065e5f698-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.964 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone3, test-zone1} 2024-03-04T16:28:08.965 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-20 zones={test-zone3, test-zone2} 2024-03-04T16:28:08.965 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.965 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.965 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.965 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:08.965 DEBUG:boto:Data: 2024-03-04T16:28:08.966 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.966 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.966 DEBUG:boto:Params: {} 2024-03-04T16:28:08.966 DEBUG:boto:Token: None 2024-03-04T16:28:08.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rhLj45KBk1/i/qDlQ3Opm1/s0Iw= 2024-03-04T16:28:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rhLj45KBk1/i/qDlQ3Opm1/s0Iw=', 'Content-Length': '0'} 2024-03-04T16:28:08.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ae4c6f9b7f58aa1-0065e5f698-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.969 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.970 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.970 DEBUG:boto:Method: HEAD 2024-03-04T16:28:08.970 DEBUG:boto:Path: /sbnvju-20/ 2024-03-04T16:28:08.970 DEBUG:boto:Data: 2024-03-04T16:28:08.970 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.970 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:08.970 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.970 DEBUG:boto:Params: {} 2024-03-04T16:28:08.970 DEBUG:boto:Token: None 2024-03-04T16:28:08.971 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/ 2024-03-04T16:28:08.971 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rhLj45KBk1/i/qDlQ3Opm1/s0Iw= 2024-03-04T16:28:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rhLj45KBk1/i/qDlQ3Opm1/s0Iw=', 'Content-Length': '0'} 2024-03-04T16:28:08.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8fe674aa86807a0-0065e5f698-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.974 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:08.974 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.974 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.974 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.974 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.974 DEBUG:boto:Method: GET 2024-03-04T16:28:08.974 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.975 DEBUG:boto:Data: 2024-03-04T16:28:08.975 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.975 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.975 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.975 DEBUG:boto:Params: {} 2024-03-04T16:28:08.975 DEBUG:boto:Token: None 2024-03-04T16:28:08.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/?versions 2024-03-04T16:28:08.975 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8= 2024-03-04T16:28:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8=', 'Content-Length': '0'} 2024-03-04T16:28:08.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52706dbd5cd5699-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.978 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:08.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:08.979 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.979 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.980 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.980 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.980 DEBUG:boto:Method: GET 2024-03-04T16:28:08.980 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.980 DEBUG:boto:Data: 2024-03-04T16:28:08.981 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.981 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:08.981 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.981 DEBUG:boto:Params: {} 2024-03-04T16:28:08.981 DEBUG:boto:Token: None 2024-03-04T16:28:08.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/?versions 2024-03-04T16:28:08.981 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8= 2024-03-04T16:28:08.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8=', 'Content-Length': '0'} 2024-03-04T16:28:08.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee44d61b9979bcfc-0065e5f698-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.985 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:08.986 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.986 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.986 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.986 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.986 DEBUG:boto:Method: GET 2024-03-04T16:28:08.986 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.986 DEBUG:boto:Data: 2024-03-04T16:28:08.986 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.986 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.986 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.987 DEBUG:boto:Params: {} 2024-03-04T16:28:08.987 DEBUG:boto:Token: None 2024-03-04T16:28:08.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/?versions 2024-03-04T16:28:08.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8= 2024-03-04T16:28:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8=', 'Content-Length': '0'} 2024-03-04T16:28:08.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d534b48d99be6e5-0065e5f698-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.990 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.990 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:08.990 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:08.991 DEBUG:boto:path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.991 DEBUG:boto:auth_path=/sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.991 DEBUG:boto:Method: GET 2024-03-04T16:28:08.991 DEBUG:boto:Path: /sbnvju-20/?versions&max-keys=999 2024-03-04T16:28:08.991 DEBUG:boto:Data: 2024-03-04T16:28:08.991 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.991 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:08.991 DEBUG:boto:Port: 8000 2024-03-04T16:28:08.991 DEBUG:boto:Params: {} 2024-03-04T16:28:08.991 DEBUG:boto:Token: None 2024-03-04T16:28:08.992 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/?versions 2024-03-04T16:28:08.992 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8= 2024-03-04T16:28:08.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KqXynxaGcnHi40p0AFpZ5yOhTI8=', 'Content-Length': '0'} 2024-03-04T16:28:08.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb71031d035b989-0065e5f698-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:08 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:08.996 DEBUG:boto:b'sbnvju-20999falsedummyNwZAMD4LoB7SUvsJQ11zbKaCnm1yoDNtrue2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1m5GTMFhrhMDR7re8TsEu3RZCBx1GaxTtrue2024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2gzeRjnEJToM8t7DqOt8LPpAi9HXDJYetrue2024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Mapt42bYytk2I8GlqJwJEryhbx-SVMutrue2024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:08.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:08.997 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:08.997 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:08.997 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.997 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.997 DEBUG:boto:Method: GET 2024-03-04T16:28:08.997 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.997 DEBUG:boto:Data: 2024-03-04T16:28:08.998 DEBUG:boto:Headers: {} 2024-03-04T16:28:08.998 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:08.998 DEBUG:boto:Port: 8001 2024-03-04T16:28:08.998 DEBUG:boto:Params: {} 2024-03-04T16:28:08.998 DEBUG:boto:Token: None 2024-03-04T16:28:08.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:08 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:08.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s= 2024-03-04T16:28:08.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:08 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lWtjuRi7/Zs5PBNnlhw7WGRiZ0s=', 'Content-Length': '0'} 2024-03-04T16:28:09.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004651d43d3343df5b-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.001 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.001 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.001 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.001 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.002 DEBUG:boto:Method: GET 2024-03-04T16:28:09.002 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.003 DEBUG:boto:Data: 2024-03-04T16:28:09.003 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.003 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.003 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.003 DEBUG:boto:Params: {} 2024-03-04T16:28:09.003 DEBUG:boto:Token: None 2024-03-04T16:28:09.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.003 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo= 2024-03-04T16:28:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo=', 'Content-Length': '0'} 2024-03-04T16:28:09.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009572e5287e7c5c0b-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.006 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.006 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.006 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.006 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.007 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.008 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.008 DEBUG:boto:Data: 2024-03-04T16:28:09.008 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.008 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.008 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.008 DEBUG:boto:Params: {} 2024-03-04T16:28:09.008 DEBUG:boto:Token: None 2024-03-04T16:28:09.008 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.009 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YlxWxgfRhwpeeLCBde1HwC4J5+I= 2024-03-04T16:28:09.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YlxWxgfRhwpeeLCBde1HwC4J5+I=', 'Content-Length': '0'} 2024-03-04T16:28:09.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034c1b2cbde7ccc61-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.012 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.012 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.012 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.012 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.013 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.013 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.013 DEBUG:boto:Data: 2024-03-04T16:28:09.013 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.013 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.013 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.013 DEBUG:boto:Params: {} 2024-03-04T16:28:09.013 DEBUG:boto:Token: None 2024-03-04T16:28:09.014 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YlxWxgfRhwpeeLCBde1HwC4J5+I= 2024-03-04T16:28:09.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YlxWxgfRhwpeeLCBde1HwC4J5+I=', 'Content-Length': '0'} 2024-03-04T16:28:09.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab76d9106170ed56-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:09.017 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.018 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.018 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.018 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.018 DEBUG:boto:Method: GET 2024-03-04T16:28:09.018 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.018 DEBUG:boto:Data: 2024-03-04T16:28:09.018 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.018 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.018 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.018 DEBUG:boto:Params: {} 2024-03-04T16:28:09.019 DEBUG:boto:Token: None 2024-03-04T16:28:09.019 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.019 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo= 2024-03-04T16:28:09.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo=', 'Content-Length': '0'} 2024-03-04T16:28:09.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f47f5804d6ddcd5f-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.021 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.021 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.022 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.022 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.022 DEBUG:boto:Method: GET 2024-03-04T16:28:09.022 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.022 DEBUG:boto:Data: 2024-03-04T16:28:09.022 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.022 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.023 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.023 DEBUG:boto:Params: {} 2024-03-04T16:28:09.023 DEBUG:boto:Token: None 2024-03-04T16:28:09.023 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.023 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo= 2024-03-04T16:28:09.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo=', 'Content-Length': '0'} 2024-03-04T16:28:09.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f86af6368b0fc06-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.025 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.025 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.026 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.026 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.026 DEBUG:boto:Method: GET 2024-03-04T16:28:09.026 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.026 DEBUG:boto:Data: 2024-03-04T16:28:09.026 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.026 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.026 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.026 DEBUG:boto:Params: {} 2024-03-04T16:28:09.026 DEBUG:boto:Token: None 2024-03-04T16:28:09.027 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.027 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043575b12aadc1684-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.072 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.073 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.073 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.073 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.073 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.073 DEBUG:boto:Method: GET 2024-03-04T16:28:09.073 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.073 DEBUG:boto:Data: 2024-03-04T16:28:09.074 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.074 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.074 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.074 DEBUG:boto:Params: {} 2024-03-04T16:28:09.074 DEBUG:boto:Token: None 2024-03-04T16:28:09.074 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.075 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b786bb66fedaaa36-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.077 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.078 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.078 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.078 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.078 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:09.078 DEBUG:boto:Data: 2024-03-04T16:28:09.079 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.079 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.079 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.079 DEBUG:boto:Params: {} 2024-03-04T16:28:09.079 DEBUG:boto:Token: None 2024-03-04T16:28:09.079 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy 2024-03-04T16:28:09.079 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5woY5R9c3mhuJQI7q0KVOHadvvc= 2024-03-04T16:28:09.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5woY5R9c3mhuJQI7q0KVOHadvvc=', 'Content-Length': '0'} 2024-03-04T16:28:09.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000236e756243ffbbf7-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.082 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.082 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.082 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.083 DEBUG:boto:Path: /sbnvju-20/dummy 2024-03-04T16:28:09.083 DEBUG:boto:Data: 2024-03-04T16:28:09.083 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.083 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.083 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.083 DEBUG:boto:Params: {} 2024-03-04T16:28:09.083 DEBUG:boto:Token: None 2024-03-04T16:28:09.084 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy 2024-03-04T16:28:09.084 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5woY5R9c3mhuJQI7q0KVOHadvvc= 2024-03-04T16:28:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5woY5R9c3mhuJQI7q0KVOHadvvc=', 'Content-Length': '0'} 2024-03-04T16:28:09.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045bc0195dd2943d6-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:09.086 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.087 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.087 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.087 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.087 DEBUG:boto:Method: GET 2024-03-04T16:28:09.087 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.087 DEBUG:boto:Data: 2024-03-04T16:28:09.087 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.087 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.087 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.087 DEBUG:boto:Params: {} 2024-03-04T16:28:09.088 DEBUG:boto:Token: None 2024-03-04T16:28:09.088 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.088 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo= 2024-03-04T16:28:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo=', 'Content-Length': '0'} 2024-03-04T16:28:09.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d285b81a47e02f97-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.090 DEBUG:boto:path=/sbnvju-20/dummy 2024-03-04T16:28:09.090 DEBUG:boto:auth_path=/sbnvju-20/dummy 2024-03-04T16:28:09.091 DEBUG:boto:path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.091 DEBUG:boto:auth_path=/sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.091 DEBUG:boto:Method: GET 2024-03-04T16:28:09.091 DEBUG:boto:Path: /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.091 DEBUG:boto:Data: 2024-03-04T16:28:09.091 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.091 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.091 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.091 DEBUG:boto:Params: {} 2024-03-04T16:28:09.092 DEBUG:boto:Token: None 2024-03-04T16:28:09.092 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/dummy?versionId=NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN 2024-03-04T16:28:09.092 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo= 2024-03-04T16:28:09.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BQhqjU4Y6Tg32AcJluJ6kHFkLPo=', 'Content-Length': '0'} 2024-03-04T16:28:09.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:24:45 GMT'), ('x-amz-version-id', 'NwZAMD4LoB7SUvsJQ11zbKaCnm1yoDN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077ca7195b8a65a7a-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.094 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.094 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.094 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.094 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.095 DEBUG:boto:Method: GET 2024-03-04T16:28:09.095 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.095 DEBUG:boto:Data: 2024-03-04T16:28:09.095 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.095 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.095 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.095 DEBUG:boto:Params: {} 2024-03-04T16:28:09.095 DEBUG:boto:Token: None 2024-03-04T16:28:09.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.095 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f842f52586d0de20-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.098 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.099 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.099 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.099 DEBUG:boto:path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.099 DEBUG:boto:auth_path=/sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.099 DEBUG:boto:Method: GET 2024-03-04T16:28:09.099 DEBUG:boto:Path: /sbnvju-20/?prefix=dummy 2024-03-04T16:28:09.099 DEBUG:boto:Data: 2024-03-04T16:28:09.099 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.099 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.099 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.100 DEBUG:boto:Params: {} 2024-03-04T16:28:09.100 DEBUG:boto:Token: None 2024-03-04T16:28:09.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.100 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31e759f7d01f73c-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.103 DEBUG:boto:b'sbnvju-20dummy1000falsedummy2024-03-04T16:24:45.206Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:09.104 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.104 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.104 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.104 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.104 DEBUG:boto:Method: GET 2024-03-04T16:28:09.104 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.104 DEBUG:boto:Data: 2024-03-04T16:28:09.105 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.105 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.105 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.105 DEBUG:boto:Params: {} 2024-03-04T16:28:09.105 DEBUG:boto:Token: None 2024-03-04T16:28:09.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.105 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eddbed0dcbe4762a-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.148 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.148 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.148 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.148 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.149 DEBUG:boto:Method: GET 2024-03-04T16:28:09.149 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.149 DEBUG:boto:Data: 2024-03-04T16:28:09.149 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.149 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.149 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.149 DEBUG:boto:Params: {} 2024-03-04T16:28:09.149 DEBUG:boto:Token: None 2024-03-04T16:28:09.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b9308db960b3ca8-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.152 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.152 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.152 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.152 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.152 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.152 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.152 DEBUG:boto:Data: 2024-03-04T16:28:09.152 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.152 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.153 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.153 DEBUG:boto:Params: {} 2024-03-04T16:28:09.153 DEBUG:boto:Token: None 2024-03-04T16:28:09.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.153 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KV8Q6r6KiWoOWoZcBVBYLC765Xg= 2024-03-04T16:28:09.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KV8Q6r6KiWoOWoZcBVBYLC765Xg=', 'Content-Length': '0'} 2024-03-04T16:28:09.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670cb7ca4bbb3055-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.155 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.155 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.155 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.155 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.155 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.155 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.156 DEBUG:boto:Data: 2024-03-04T16:28:09.156 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.156 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.156 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.156 DEBUG:boto:Params: {} 2024-03-04T16:28:09.156 DEBUG:boto:Token: None 2024-03-04T16:28:09.156 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.156 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KV8Q6r6KiWoOWoZcBVBYLC765Xg= 2024-03-04T16:28:09.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KV8Q6r6KiWoOWoZcBVBYLC765Xg=', 'Content-Length': '0'} 2024-03-04T16:28:09.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e195940940011f20-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:09.158 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.158 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.158 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.159 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.159 DEBUG:boto:Method: GET 2024-03-04T16:28:09.159 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.159 DEBUG:boto:Data: 2024-03-04T16:28:09.159 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.159 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.159 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.159 DEBUG:boto:Params: {} 2024-03-04T16:28:09.160 DEBUG:boto:Token: None 2024-03-04T16:28:09.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.160 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003794a2d220494302-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.204 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.204 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.204 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.204 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.205 DEBUG:boto:Method: GET 2024-03-04T16:28:09.205 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.205 DEBUG:boto:Data: 2024-03-04T16:28:09.205 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.205 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.205 DEBUG:boto:Params: {} 2024-03-04T16:28:09.205 DEBUG:boto:Token: None 2024-03-04T16:28:09.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c142c86a09df835d-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.248 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.248 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.248 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.249 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.249 DEBUG:boto:Method: GET 2024-03-04T16:28:09.249 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.249 DEBUG:boto:Data: 2024-03-04T16:28:09.249 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.249 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.249 DEBUG:boto:Params: {} 2024-03-04T16:28:09.249 DEBUG:boto:Token: None 2024-03-04T16:28:09.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.250 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008568d001b2bf3951-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.300 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.301 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.301 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.301 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.301 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.301 DEBUG:boto:Method: GET 2024-03-04T16:28:09.301 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.301 DEBUG:boto:Data: 2024-03-04T16:28:09.301 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.301 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.301 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.301 DEBUG:boto:Params: {} 2024-03-04T16:28:09.302 DEBUG:boto:Token: None 2024-03-04T16:28:09.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba503ab4e96f1cd-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.305 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.306 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.306 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.306 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.306 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.306 DEBUG:boto:Data: 2024-03-04T16:28:09.306 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.306 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.306 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.306 DEBUG:boto:Params: {} 2024-03-04T16:28:09.307 DEBUG:boto:Token: None 2024-03-04T16:28:09.307 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Via9/sr5pol5OLbgt4HdtXk8LE= 2024-03-04T16:28:09.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Via9/sr5pol5OLbgt4HdtXk8LE=', 'Content-Length': '0'} 2024-03-04T16:28:09.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7ec6cde86100d40-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.310 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.310 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.310 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.310 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.310 DEBUG:boto:Data: 2024-03-04T16:28:09.310 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.310 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.310 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.310 DEBUG:boto:Params: {} 2024-03-04T16:28:09.311 DEBUG:boto:Token: None 2024-03-04T16:28:09.311 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Via9/sr5pol5OLbgt4HdtXk8LE= 2024-03-04T16:28:09.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Via9/sr5pol5OLbgt4HdtXk8LE=', 'Content-Length': '0'} 2024-03-04T16:28:09.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061e2c1dad4c0854e-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:09.314 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.314 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.314 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.314 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.314 DEBUG:boto:Method: GET 2024-03-04T16:28:09.314 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.314 DEBUG:boto:Data: 2024-03-04T16:28:09.315 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.315 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.315 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.315 DEBUG:boto:Params: {} 2024-03-04T16:28:09.315 DEBUG:boto:Token: None 2024-03-04T16:28:09.315 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028320972394cbd7a-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.361 DEBUG:boto:path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.361 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1 2024-03-04T16:28:09.361 DEBUG:boto:path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.361 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.361 DEBUG:boto:Method: GET 2024-03-04T16:28:09.361 DEBUG:boto:Path: /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.361 DEBUG:boto:Data: 2024-03-04T16:28:09.362 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.362 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.362 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.362 DEBUG:boto:Params: {} 2024-03-04T16:28:09.362 DEBUG:boto:Token: None 2024-03-04T16:28:09.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone1?versionId=m5GTMFhrhMDR7re8TsEu3RZCBx1GaxT 2024-03-04T16:28:09.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI= 2024-03-04T16:28:09.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WsMxh6CLfyhEaiuRu66ph/E25PI=', 'Content-Length': '0'} 2024-03-04T16:28:09.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'm5GTMFhrhMDR7re8TsEu3RZCBx1GaxT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a5138a8cb7948e8-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.408 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.408 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.408 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.408 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.409 DEBUG:boto:Method: GET 2024-03-04T16:28:09.409 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.409 DEBUG:boto:Data: 2024-03-04T16:28:09.409 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.409 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.409 DEBUG:boto:Params: {} 2024-03-04T16:28:09.409 DEBUG:boto:Token: None 2024-03-04T16:28:09.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064193f3a5aa8a3ae-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.456 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.457 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.457 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.458 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.459 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.459 DEBUG:boto:Method: GET 2024-03-04T16:28:09.460 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone1 2024-03-04T16:28:09.460 DEBUG:boto:Data: 2024-03-04T16:28:09.460 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.460 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.460 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.461 DEBUG:boto:Params: {} 2024-03-04T16:28:09.461 DEBUG:boto:Token: None 2024-03-04T16:28:09.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a8b0f8a953f37d-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.464 DEBUG:boto:b'sbnvju-20obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.099Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:09.465 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.465 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.465 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.466 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.466 DEBUG:boto:Method: GET 2024-03-04T16:28:09.466 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.466 DEBUG:boto:Data: 2024-03-04T16:28:09.466 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.466 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.466 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.466 DEBUG:boto:Params: {} 2024-03-04T16:28:09.466 DEBUG:boto:Token: None 2024-03-04T16:28:09.467 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.467 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6aba82c24afe947-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.512 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.512 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.512 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.512 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.513 DEBUG:boto:Method: GET 2024-03-04T16:28:09.513 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.513 DEBUG:boto:Data: 2024-03-04T16:28:09.513 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.513 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.513 DEBUG:boto:Params: {} 2024-03-04T16:28:09.513 DEBUG:boto:Token: None 2024-03-04T16:28:09.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c693b302fb55cb9-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.516 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.516 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.516 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.516 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.516 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.517 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.517 DEBUG:boto:Data: 2024-03-04T16:28:09.517 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.517 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.517 DEBUG:boto:Params: {} 2024-03-04T16:28:09.517 DEBUG:boto:Token: None 2024-03-04T16:28:09.517 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TaQAi0ABeQPOIuKrvMXdUxx7Suo= 2024-03-04T16:28:09.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TaQAi0ABeQPOIuKrvMXdUxx7Suo=', 'Content-Length': '0'} 2024-03-04T16:28:09.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e9b0c69902be3a0-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.519 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.520 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.520 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.520 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.520 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.520 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.520 DEBUG:boto:Data: 2024-03-04T16:28:09.520 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.520 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.520 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.520 DEBUG:boto:Params: {} 2024-03-04T16:28:09.521 DEBUG:boto:Token: None 2024-03-04T16:28:09.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TaQAi0ABeQPOIuKrvMXdUxx7Suo= 2024-03-04T16:28:09.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TaQAi0ABeQPOIuKrvMXdUxx7Suo=', 'Content-Length': '0'} 2024-03-04T16:28:09.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000274cc48724c28fa1-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:09.523 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.523 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.523 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.523 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.523 DEBUG:boto:Method: GET 2024-03-04T16:28:09.523 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.523 DEBUG:boto:Data: 2024-03-04T16:28:09.523 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.524 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.524 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.524 DEBUG:boto:Params: {} 2024-03-04T16:28:09.524 DEBUG:boto:Token: None 2024-03-04T16:28:09.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cc84570cd6f5206-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.568 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.568 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.568 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.569 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.569 DEBUG:boto:Method: GET 2024-03-04T16:28:09.569 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.569 DEBUG:boto:Data: 2024-03-04T16:28:09.569 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.569 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.569 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.569 DEBUG:boto:Params: {} 2024-03-04T16:28:09.569 DEBUG:boto:Token: None 2024-03-04T16:28:09.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070120fcc234fb02a-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.573 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.573 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.573 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.573 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.573 DEBUG:boto:Method: GET 2024-03-04T16:28:09.573 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.573 DEBUG:boto:Data: 2024-03-04T16:28:09.573 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.573 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.573 DEBUG:boto:Params: {} 2024-03-04T16:28:09.574 DEBUG:boto:Token: None 2024-03-04T16:28:09.574 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116b299b5f44ecf7-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.620 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.620 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.621 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.621 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.621 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.621 DEBUG:boto:Method: GET 2024-03-04T16:28:09.621 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.621 DEBUG:boto:Data: 2024-03-04T16:28:09.621 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.621 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.621 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.621 DEBUG:boto:Params: {} 2024-03-04T16:28:09.622 DEBUG:boto:Token: None 2024-03-04T16:28:09.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b38bc0d06a1b263-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.625 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.625 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.625 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.625 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.625 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.625 DEBUG:boto:Data: 2024-03-04T16:28:09.625 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.626 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.626 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.626 DEBUG:boto:Params: {} 2024-03-04T16:28:09.626 DEBUG:boto:Token: None 2024-03-04T16:28:09.626 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SVr/bv2RZsfjPkliKg618hjDoMU= 2024-03-04T16:28:09.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SVr/bv2RZsfjPkliKg618hjDoMU=', 'Content-Length': '0'} 2024-03-04T16:28:09.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdc5ee20a4148863-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.629 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.629 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.629 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.629 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.629 DEBUG:boto:Data: 2024-03-04T16:28:09.629 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.630 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.630 DEBUG:boto:Params: {} 2024-03-04T16:28:09.630 DEBUG:boto:Token: None 2024-03-04T16:28:09.630 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SVr/bv2RZsfjPkliKg618hjDoMU= 2024-03-04T16:28:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SVr/bv2RZsfjPkliKg618hjDoMU=', 'Content-Length': '0'} 2024-03-04T16:28:09.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9d8ce7b4a580015-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:09.634 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.634 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.634 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.634 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.634 DEBUG:boto:Method: GET 2024-03-04T16:28:09.634 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.634 DEBUG:boto:Data: 2024-03-04T16:28:09.634 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.635 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.635 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.635 DEBUG:boto:Params: {} 2024-03-04T16:28:09.635 DEBUG:boto:Token: None 2024-03-04T16:28:09.635 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.635 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060f8a391c3946844-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.684 DEBUG:boto:path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.684 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2 2024-03-04T16:28:09.685 DEBUG:boto:path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.685 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.685 DEBUG:boto:Method: GET 2024-03-04T16:28:09.685 DEBUG:boto:Path: /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.685 DEBUG:boto:Data: 2024-03-04T16:28:09.685 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.685 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.685 DEBUG:boto:Params: {} 2024-03-04T16:28:09.686 DEBUG:boto:Token: None 2024-03-04T16:28:09.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone2?versionId=gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe 2024-03-04T16:28:09.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0= 2024-03-04T16:28:09.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FWzm9FD+2MJAq7POhJOV9RR76Z0=', 'Content-Length': '0'} 2024-03-04T16:28:09.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'gzeRjnEJToM8t7DqOt8LPpAi9HXDJYe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005128c19350d7c8fb-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.688 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.689 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.689 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.689 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.689 DEBUG:boto:Method: GET 2024-03-04T16:28:09.689 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.689 DEBUG:boto:Data: 2024-03-04T16:28:09.689 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.689 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.689 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.689 DEBUG:boto:Params: {} 2024-03-04T16:28:09.690 DEBUG:boto:Token: None 2024-03-04T16:28:09.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a055be743cb448-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.736 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.737 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.737 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.737 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.737 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.737 DEBUG:boto:Method: GET 2024-03-04T16:28:09.737 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone2 2024-03-04T16:28:09.737 DEBUG:boto:Data: 2024-03-04T16:28:09.737 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.738 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.738 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.738 DEBUG:boto:Params: {} 2024-03-04T16:28:09.738 DEBUG:boto:Token: None 2024-03-04T16:28:09.738 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.738 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078553e74679b6ed9-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.741 DEBUG:boto:b'sbnvju-20obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.196Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:09.742 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.742 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.742 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.742 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.742 DEBUG:boto:Method: GET 2024-03-04T16:28:09.742 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.742 DEBUG:boto:Data: 2024-03-04T16:28:09.742 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.742 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.742 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.743 DEBUG:boto:Params: {} 2024-03-04T16:28:09.743 DEBUG:boto:Token: None 2024-03-04T16:28:09.743 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.743 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062d82b08308335f1-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.745 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.746 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.746 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.746 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.746 DEBUG:boto:Method: GET 2024-03-04T16:28:09.746 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.746 DEBUG:boto:Data: 2024-03-04T16:28:09.746 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.746 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.746 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.746 DEBUG:boto:Params: {} 2024-03-04T16:28:09.747 DEBUG:boto:Token: None 2024-03-04T16:28:09.747 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.747 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000401292f1069114fb-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.750 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.750 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.750 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.750 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.750 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.750 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.750 DEBUG:boto:Data: 2024-03-04T16:28:09.751 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.751 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.751 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.751 DEBUG:boto:Params: {} 2024-03-04T16:28:09.751 DEBUG:boto:Token: None 2024-03-04T16:28:09.751 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.752 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C/XpttpZu8w7S1Kcb5UVbIla87Y= 2024-03-04T16:28:09.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C/XpttpZu8w7S1Kcb5UVbIla87Y=', 'Content-Length': '0'} 2024-03-04T16:28:09.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4c289e9acd1102f-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.754 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.754 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.754 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.754 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.754 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.754 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.754 DEBUG:boto:Data: 2024-03-04T16:28:09.754 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.755 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.755 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.755 DEBUG:boto:Params: {} 2024-03-04T16:28:09.755 DEBUG:boto:Token: None 2024-03-04T16:28:09.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C/XpttpZu8w7S1Kcb5UVbIla87Y= 2024-03-04T16:28:09.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C/XpttpZu8w7S1Kcb5UVbIla87Y=', 'Content-Length': '0'} 2024-03-04T16:28:09.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079d330b4c8fa7886-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:09.758 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.758 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.758 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.759 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.759 DEBUG:boto:Method: GET 2024-03-04T16:28:09.759 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.759 DEBUG:boto:Data: 2024-03-04T16:28:09.759 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.759 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.759 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.759 DEBUG:boto:Params: {} 2024-03-04T16:28:09.760 DEBUG:boto:Token: None 2024-03-04T16:28:09.760 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.760 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb9a127811ffcdf4-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.763 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.763 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.763 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.763 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.764 DEBUG:boto:Method: GET 2024-03-04T16:28:09.764 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.764 DEBUG:boto:Data: 2024-03-04T16:28:09.764 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.764 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.764 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.764 DEBUG:boto:Params: {} 2024-03-04T16:28:09.765 DEBUG:boto:Token: None 2024-03-04T16:28:09.765 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7f27be7c1835987-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.808 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.808 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.808 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.808 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.809 DEBUG:boto:Method: GET 2024-03-04T16:28:09.809 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.809 DEBUG:boto:Data: 2024-03-04T16:28:09.809 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.809 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.809 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.809 DEBUG:boto:Params: {} 2024-03-04T16:28:09.809 DEBUG:boto:Token: None 2024-03-04T16:28:09.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.810 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c03ac291a32ec4-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.812 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.813 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.813 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.813 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.813 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.813 DEBUG:boto:Method: GET 2024-03-04T16:28:09.813 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.813 DEBUG:boto:Data: 2024-03-04T16:28:09.814 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.814 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.814 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.814 DEBUG:boto:Params: {} 2024-03-04T16:28:09.814 DEBUG:boto:Token: None 2024-03-04T16:28:09.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c969b495501ebe8-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.860 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.861 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.861 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.861 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.861 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.861 DEBUG:boto:Data: 2024-03-04T16:28:09.861 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.861 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.861 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.861 DEBUG:boto:Params: {} 2024-03-04T16:28:09.862 DEBUG:boto:Token: None 2024-03-04T16:28:09.862 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uCWxSbAdGPYjy/kdlrGfh3nLY70= 2024-03-04T16:28:09.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uCWxSbAdGPYjy/kdlrGfh3nLY70=', 'Content-Length': '0'} 2024-03-04T16:28:09.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016f150ae9e0af71f-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.865 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.865 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.865 DEBUG:boto:Method: HEAD 2024-03-04T16:28:09.865 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.865 DEBUG:boto:Data: 2024-03-04T16:28:09.865 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.865 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.865 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.865 DEBUG:boto:Params: {} 2024-03-04T16:28:09.866 DEBUG:boto:Token: None 2024-03-04T16:28:09.866 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.866 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uCWxSbAdGPYjy/kdlrGfh3nLY70= 2024-03-04T16:28:09.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uCWxSbAdGPYjy/kdlrGfh3nLY70=', 'Content-Length': '0'} 2024-03-04T16:28:09.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006372c7f77b83224b-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:09.869 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.869 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.869 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.869 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.869 DEBUG:boto:Method: GET 2024-03-04T16:28:09.869 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.869 DEBUG:boto:Data: 2024-03-04T16:28:09.869 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.869 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.869 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.869 DEBUG:boto:Params: {} 2024-03-04T16:28:09.870 DEBUG:boto:Token: None 2024-03-04T16:28:09.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.870 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000746f98294320f82c-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.912 DEBUG:boto:path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.912 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3 2024-03-04T16:28:09.912 DEBUG:boto:path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.912 DEBUG:boto:auth_path=/sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.913 DEBUG:boto:Method: GET 2024-03-04T16:28:09.913 DEBUG:boto:Path: /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.913 DEBUG:boto:Data: 2024-03-04T16:28:09.913 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.913 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.913 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.913 DEBUG:boto:Params: {} 2024-03-04T16:28:09.913 DEBUG:boto:Token: None 2024-03-04T16:28:09.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/obj-test-zone3?versionId=Mapt42bYytk2I8GlqJwJEryhbx-SVMu 2024-03-04T16:28:09.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74= 2024-03-04T16:28:09.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k65nEykh+wc7NiCioc1EDWj+x74=', 'Content-Length': '0'} 2024-03-04T16:28:09.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'Mapt42bYytk2I8GlqJwJEryhbx-SVMu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-20:fc294220-5038-4f25-aeee-cac2c5838729.4341.29'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e771dc602acb732-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.956 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.956 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.956 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.956 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.957 DEBUG:boto:Method: GET 2024-03-04T16:28:09.957 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.957 DEBUG:boto:Data: 2024-03-04T16:28:09.957 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:09.957 DEBUG:boto:Port: 8001 2024-03-04T16:28:09.957 DEBUG:boto:Params: {} 2024-03-04T16:28:09.957 DEBUG:boto:Token: None 2024-03-04T16:28:09.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba409664e3a3f46e-0065e5f699-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:09.961 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:09.961 DEBUG:boto:path=/sbnvju-20/ 2024-03-04T16:28:09.961 DEBUG:boto:auth_path=/sbnvju-20/ 2024-03-04T16:28:09.961 DEBUG:boto:path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.962 DEBUG:boto:auth_path=/sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.962 DEBUG:boto:Method: GET 2024-03-04T16:28:09.962 DEBUG:boto:Path: /sbnvju-20/?prefix=obj-test-zone3 2024-03-04T16:28:09.962 DEBUG:boto:Data: 2024-03-04T16:28:09.962 DEBUG:boto:Headers: {} 2024-03-04T16:28:09.962 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:09.962 DEBUG:boto:Port: 8000 2024-03-04T16:28:09.962 DEBUG:boto:Params: {} 2024-03-04T16:28:09.962 DEBUG:boto:Token: None 2024-03-04T16:28:09.962 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:09 GMT /sbnvju-20/ 2024-03-04T16:28:09.963 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds= 2024-03-04T16:28:09.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+7i+4P2ZcO4gRgn/oHDFDRIi1Ds=', 'Content-Length': '0'} 2024-03-04T16:28:09.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678d69c64c0297b1-0065e5f699-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.008 DEBUG:boto:b'sbnvju-20obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.294Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-20 zones={test-zone3, test-zone2} 2024-03-04T16:28:10.009 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-21 2024-03-04T16:28:10.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.010 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.125 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:10.125+0000 7effe3f4b800 1 bucket sync caught up with source: 2024-03-04T16:28:10.125 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000011.70.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:28:10.125 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000011.70.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:28:10.125 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:10.125+0000 7effe3f4b800 0 bucket checkpoint complete 2024-03-04T16:28:10.144 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-21 2024-03-04T16:28:10.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.145 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.207 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.207 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.262+0000 7fab29e85800 1 bucket sync caught up with source: 2024-03-04T16:28:10.263 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000011.70.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:28:10.263 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000011.70.6, , 00000000003.36.10, , 00000000007.31.10, , , , , ] 2024-03-04T16:28:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.262+0000 7fab29e85800 0 bucket checkpoint complete 2024-03-04T16:28:10.282 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-21 2024-03-04T16:28:10.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.282 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.397+0000 7ff61961b800 1 bucket sync caught up with source: 2024-03-04T16:28:10.397 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.52.10, , 00000000011.47.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:28:10.397 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.52.10, , 00000000011.47.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:28:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.397+0000 7ff61961b800 0 bucket checkpoint complete 2024-03-04T16:28:10.416 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-21 2024-03-04T16:28:10.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.417 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.522+0000 7f1043c8c800 1 bucket sync caught up with source: 2024-03-04T16:28:10.523 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.52.10, , 00000000011.47.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:28:10.523 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.52.10, , 00000000011.47.6, , 00000000006.28.10, , , , , ] 2024-03-04T16:28:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.522+0000 7f1043c8c800 0 bucket checkpoint complete 2024-03-04T16:28:10.541 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-21 2024-03-04T16:28:10.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.542 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.646+0000 7fa6b4239800 1 bucket sync caught up with source: 2024-03-04T16:28:10.647 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, 00000000003.53.10, , 00000000003.31.10, , 00000000016.48.6, , , , , ] 2024-03-04T16:28:10.647 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, 00000000003.53.10, , 00000000003.31.10, , 00000000016.48.6, , , , , ] 2024-03-04T16:28:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:10.646+0000 7fa6b4239800 0 bucket checkpoint complete 2024-03-04T16:28:10.665 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-21 2024-03-04T16:28:10.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:10.666 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:10.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:10.772+0000 7f5a31921800 1 bucket sync caught up with source: 2024-03-04T16:28:10.772 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, 00000000003.53.10, , 00000000003.31.10, , 00000000016.48.6, , , , , ] 2024-03-04T16:28:10.772 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, 00000000003.53.10, , 00000000003.31.10, , 00000000016.48.6, , , , , ] 2024-03-04T16:28:10.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:10.772+0000 7f5a31921800 0 bucket checkpoint complete 2024-03-04T16:28:10.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone2, test-zone1} 2024-03-04T16:28:10.790 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.790 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.790 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.790 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:10.790 DEBUG:boto:Data: 2024-03-04T16:28:10.790 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.790 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.791 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.791 DEBUG:boto:Params: {} 2024-03-04T16:28:10.791 DEBUG:boto:Token: None 2024-03-04T16:28:10.791 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wCdGQcFc7qg6ZC2jcq9xc2iOG/I= 2024-03-04T16:28:10.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wCdGQcFc7qg6ZC2jcq9xc2iOG/I=', 'Content-Length': '0'} 2024-03-04T16:28:10.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aeeabee8af8ee950-0065e5f69a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.794 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.794 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.794 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.794 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:10.794 DEBUG:boto:Data: 2024-03-04T16:28:10.794 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.794 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.794 DEBUG:boto:Params: {} 2024-03-04T16:28:10.794 DEBUG:boto:Token: None 2024-03-04T16:28:10.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wCdGQcFc7qg6ZC2jcq9xc2iOG/I= 2024-03-04T16:28:10.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wCdGQcFc7qg6ZC2jcq9xc2iOG/I=', 'Content-Length': '0'} 2024-03-04T16:28:10.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cfa58d3fbd288a7-0065e5f69a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.797 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:10.798 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.798 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.798 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.798 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.798 DEBUG:boto:Method: GET 2024-03-04T16:28:10.798 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.799 DEBUG:boto:Data: 2024-03-04T16:28:10.799 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.799 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.799 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.799 DEBUG:boto:Params: {} 2024-03-04T16:28:10.799 DEBUG:boto:Token: None 2024-03-04T16:28:10.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/?versions 2024-03-04T16:28:10.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4= 2024-03-04T16:28:10.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4=', 'Content-Length': '0'} 2024-03-04T16:28:10.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d848104ee530922c-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.802 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.802 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:10.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:10.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:10.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:10.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:10.805 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.805 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.805 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.805 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.806 DEBUG:boto:Method: GET 2024-03-04T16:28:10.806 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.806 DEBUG:boto:Data: 2024-03-04T16:28:10.806 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.806 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.806 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.806 DEBUG:boto:Params: {} 2024-03-04T16:28:10.807 DEBUG:boto:Token: None 2024-03-04T16:28:10.807 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/?versions 2024-03-04T16:28:10.808 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4= 2024-03-04T16:28:10.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4=', 'Content-Length': '0'} 2024-03-04T16:28:10.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680f26665508fe76-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.812 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:10.813 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.813 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.813 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.813 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.813 DEBUG:boto:Method: GET 2024-03-04T16:28:10.814 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.814 DEBUG:boto:Data: 2024-03-04T16:28:10.814 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.814 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.814 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.814 DEBUG:boto:Params: {} 2024-03-04T16:28:10.814 DEBUG:boto:Token: None 2024-03-04T16:28:10.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/?versions 2024-03-04T16:28:10.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4= 2024-03-04T16:28:10.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4=', 'Content-Length': '0'} 2024-03-04T16:28:10.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccdf6deb0dd86990-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.822 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.823 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.823 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.823 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.823 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.823 DEBUG:boto:Method: GET 2024-03-04T16:28:10.823 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:10.823 DEBUG:boto:Data: 2024-03-04T16:28:10.823 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.824 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.824 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.824 DEBUG:boto:Params: {} 2024-03-04T16:28:10.824 DEBUG:boto:Token: None 2024-03-04T16:28:10.824 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/?versions 2024-03-04T16:28:10.824 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4= 2024-03-04T16:28:10.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zO1iA0AdJmpJEA2sfU8AfFfLjN4=', 'Content-Length': '0'} 2024-03-04T16:28:10.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8a9698e12344e8-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.827 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:10.828 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.828 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.828 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.828 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.828 DEBUG:boto:Method: GET 2024-03-04T16:28:10.828 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.828 DEBUG:boto:Data: 2024-03-04T16:28:10.828 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.828 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.829 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.829 DEBUG:boto:Params: {} 2024-03-04T16:28:10.829 DEBUG:boto:Token: None 2024-03-04T16:28:10.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005560107e5636de3c-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.831 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.831 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.831 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.831 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.832 DEBUG:boto:Method: GET 2024-03-04T16:28:10.832 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.832 DEBUG:boto:Data: 2024-03-04T16:28:10.832 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.832 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.832 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.832 DEBUG:boto:Params: {} 2024-03-04T16:28:10.833 DEBUG:boto:Token: None 2024-03-04T16:28:10.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1934d3a151cba48-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.836 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.836 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.836 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.837 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.837 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.837 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.837 DEBUG:boto:Data: 2024-03-04T16:28:10.837 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.837 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.837 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.837 DEBUG:boto:Params: {} 2024-03-04T16:28:10.837 DEBUG:boto:Token: None 2024-03-04T16:28:10.838 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/UgZP18HmLfzayLCWBUN/8+4FxI= 2024-03-04T16:28:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/UgZP18HmLfzayLCWBUN/8+4FxI=', 'Content-Length': '0'} 2024-03-04T16:28:10.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5bd391c1cef41d2-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.840 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.840 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.840 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.840 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.840 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.840 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.840 DEBUG:boto:Data: 2024-03-04T16:28:10.841 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.841 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.841 DEBUG:boto:Params: {} 2024-03-04T16:28:10.841 DEBUG:boto:Token: None 2024-03-04T16:28:10.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.841 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/UgZP18HmLfzayLCWBUN/8+4FxI= 2024-03-04T16:28:10.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/UgZP18HmLfzayLCWBUN/8+4FxI=', 'Content-Length': '0'} 2024-03-04T16:28:10.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000197a6fed04aafd4a-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:10.844 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.844 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.844 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.844 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.844 DEBUG:boto:Method: GET 2024-03-04T16:28:10.844 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.844 DEBUG:boto:Data: 2024-03-04T16:28:10.844 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.844 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.844 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.845 DEBUG:boto:Params: {} 2024-03-04T16:28:10.845 DEBUG:boto:Token: None 2024-03-04T16:28:10.845 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013ce2aeab98efee5-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.848 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.848 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.848 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.848 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.848 DEBUG:boto:Method: GET 2024-03-04T16:28:10.848 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.848 DEBUG:boto:Data: 2024-03-04T16:28:10.849 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.849 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.849 DEBUG:boto:Params: {} 2024-03-04T16:28:10.849 DEBUG:boto:Token: None 2024-03-04T16:28:10.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031be7d6fdb4148b2-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.858 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.858 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.858 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.859 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.859 DEBUG:boto:Method: GET 2024-03-04T16:28:10.859 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.859 DEBUG:boto:Data: 2024-03-04T16:28:10.859 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.859 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.859 DEBUG:boto:Params: {} 2024-03-04T16:28:10.859 DEBUG:boto:Token: None 2024-03-04T16:28:10.859 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI= 2024-03-04T16:28:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI=', 'Content-Length': '0'} 2024-03-04T16:28:10.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001252c64f27681447-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.862 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.862 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.863 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.863 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.863 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.863 DEBUG:boto:Method: GET 2024-03-04T16:28:10.863 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.863 DEBUG:boto:Data: 2024-03-04T16:28:10.863 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.863 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.863 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.863 DEBUG:boto:Params: {} 2024-03-04T16:28:10.864 DEBUG:boto:Token: None 2024-03-04T16:28:10.864 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.864 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI= 2024-03-04T16:28:10.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI=', 'Content-Length': '0'} 2024-03-04T16:28:10.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035fff532967672c8-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.866 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.866 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.866 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.867 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.867 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:10.867 DEBUG:boto:Data: 2024-03-04T16:28:10.867 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.867 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.867 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.867 DEBUG:boto:Params: {} 2024-03-04T16:28:10.867 DEBUG:boto:Token: None 2024-03-04T16:28:10.868 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy 2024-03-04T16:28:10.868 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZTRJWmbkxH/zEmCGSDUCgcEwKJc= 2024-03-04T16:28:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZTRJWmbkxH/zEmCGSDUCgcEwKJc=', 'Content-Length': '0'} 2024-03-04T16:28:10.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000020297f7e883d7c0-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.870 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.870 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.871 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.871 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:10.871 DEBUG:boto:Data: 2024-03-04T16:28:10.871 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.871 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.871 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.871 DEBUG:boto:Params: {} 2024-03-04T16:28:10.871 DEBUG:boto:Token: None 2024-03-04T16:28:10.872 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy 2024-03-04T16:28:10.872 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZTRJWmbkxH/zEmCGSDUCgcEwKJc= 2024-03-04T16:28:10.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZTRJWmbkxH/zEmCGSDUCgcEwKJc=', 'Content-Length': '0'} 2024-03-04T16:28:10.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ca30d33b6e4d2a6-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:10.875 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.875 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.875 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.875 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.875 DEBUG:boto:Method: GET 2024-03-04T16:28:10.875 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.875 DEBUG:boto:Data: 2024-03-04T16:28:10.875 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.875 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.875 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.875 DEBUG:boto:Params: {} 2024-03-04T16:28:10.876 DEBUG:boto:Token: None 2024-03-04T16:28:10.876 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.876 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008dd409bb80481fb8-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.878 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:10.878 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:10.878 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.878 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.878 DEBUG:boto:Method: GET 2024-03-04T16:28:10.879 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.879 DEBUG:boto:Data: 2024-03-04T16:28:10.879 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.879 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.879 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.879 DEBUG:boto:Params: {} 2024-03-04T16:28:10.879 DEBUG:boto:Token: None 2024-03-04T16:28:10.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:10.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc= 2024-03-04T16:28:10.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:khimq20F+6PPSbSsGLjYQYpL0Wc=', 'Content-Length': '0'} 2024-03-04T16:28:10.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005171b23fb9155efd-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.881 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.881 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.881 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.881 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.882 DEBUG:boto:Method: GET 2024-03-04T16:28:10.882 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.882 DEBUG:boto:Data: 2024-03-04T16:28:10.882 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.882 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.882 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.882 DEBUG:boto:Params: {} 2024-03-04T16:28:10.882 DEBUG:boto:Token: None 2024-03-04T16:28:10.883 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.883 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI= 2024-03-04T16:28:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI=', 'Content-Length': '0'} 2024-03-04T16:28:10.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964e754f98b7fd32-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.888 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.889 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:10.889 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:10.889 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.889 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.889 DEBUG:boto:Method: GET 2024-03-04T16:28:10.889 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:10.889 DEBUG:boto:Data: 2024-03-04T16:28:10.890 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.890 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.890 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.890 DEBUG:boto:Params: {} 2024-03-04T16:28:10.890 DEBUG:boto:Token: None 2024-03-04T16:28:10.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/ 2024-03-04T16:28:10.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI= 2024-03-04T16:28:10.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DrL9rZ9+rafWGBREs0Pl3RgpCvI=', 'Content-Length': '0'} 2024-03-04T16:28:10.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a672a976cd40d3-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.893 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:10.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:10.894 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.894 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.894 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.894 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.895 DEBUG:boto:Method: GET 2024-03-04T16:28:10.895 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.895 DEBUG:boto:Data: 2024-03-04T16:28:10.895 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.895 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.895 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.895 DEBUG:boto:Params: {} 2024-03-04T16:28:10.895 DEBUG:boto:Token: None 2024-03-04T16:28:10.896 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.896 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM= 2024-03-04T16:28:10.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM=', 'Content-Length': '0'} 2024-03-04T16:28:10.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000742b9cebe5e9df9f-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.940 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.941 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.942 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.942 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.942 DEBUG:boto:Method: GET 2024-03-04T16:28:10.942 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.942 DEBUG:boto:Data: 2024-03-04T16:28:10.942 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.942 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.942 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.942 DEBUG:boto:Params: {} 2024-03-04T16:28:10.943 DEBUG:boto:Token: None 2024-03-04T16:28:10.943 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.943 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM= 2024-03-04T16:28:10.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM=', 'Content-Length': '0'} 2024-03-04T16:28:10.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3c9ed0c2caf6541-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.945 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.946 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.946 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.946 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.946 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.946 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.946 DEBUG:boto:Data: 2024-03-04T16:28:10.946 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.946 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.946 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.946 DEBUG:boto:Params: {} 2024-03-04T16:28:10.947 DEBUG:boto:Token: None 2024-03-04T16:28:10.947 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.947 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+tjl6BmmROPj9J6wM9nuxUkWTKE= 2024-03-04T16:28:10.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+tjl6BmmROPj9J6wM9nuxUkWTKE=', 'Content-Length': '0'} 2024-03-04T16:28:10.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c24fe8f40f518a55-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.949 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.950 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.950 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.950 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.950 DEBUG:boto:Method: HEAD 2024-03-04T16:28:10.950 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.950 DEBUG:boto:Data: 2024-03-04T16:28:10.950 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.950 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:10.950 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.950 DEBUG:boto:Params: {} 2024-03-04T16:28:10.951 DEBUG:boto:Token: None 2024-03-04T16:28:10.951 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.951 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+tjl6BmmROPj9J6wM9nuxUkWTKE= 2024-03-04T16:28:10.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+tjl6BmmROPj9J6wM9nuxUkWTKE=', 'Content-Length': '0'} 2024-03-04T16:28:10.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000064d20f64a29b138-0065e5f69a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:10.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:10.953 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.953 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:10.953 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.953 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.954 DEBUG:boto:Method: GET 2024-03-04T16:28:10.954 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.954 DEBUG:boto:Data: 2024-03-04T16:28:10.954 DEBUG:boto:Headers: {} 2024-03-04T16:28:10.954 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:10.954 DEBUG:boto:Port: 8000 2024-03-04T16:28:10.954 DEBUG:boto:Params: {} 2024-03-04T16:28:10.954 DEBUG:boto:Token: None 2024-03-04T16:28:10.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:10 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:10.955 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM= 2024-03-04T16:28:10.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:10 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fLSNoFzCwJ18m8iN42WkarECwyM=', 'Content-Length': '0'} 2024-03-04T16:28:10.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f76aa5b297a20d04-0065e5f69a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:10 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.000 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.000 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.001 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.001 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.001 DEBUG:boto:Method: GET 2024-03-04T16:28:11.001 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.001 DEBUG:boto:Data: 2024-03-04T16:28:11.001 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.001 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.001 DEBUG:boto:Params: {} 2024-03-04T16:28:11.002 DEBUG:boto:Token: None 2024-03-04T16:28:11.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094bd9af7851dd627-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.007 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.007 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.007 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.007 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.007 DEBUG:boto:Method: GET 2024-03-04T16:28:11.007 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.007 DEBUG:boto:Data: 2024-03-04T16:28:11.007 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.008 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.008 DEBUG:boto:Params: {} 2024-03-04T16:28:11.008 DEBUG:boto:Token: None 2024-03-04T16:28:11.008 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.008 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000893c43ef82318a0a-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.052 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.053 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.053 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.053 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.053 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.053 DEBUG:boto:Method: GET 2024-03-04T16:28:11.053 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.053 DEBUG:boto:Data: 2024-03-04T16:28:11.053 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.053 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.053 DEBUG:boto:Params: {} 2024-03-04T16:28:11.054 DEBUG:boto:Token: None 2024-03-04T16:28:11.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.054 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ae58416591bf98-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.056 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.057 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.057 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.057 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.057 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.057 DEBUG:boto:Data: 2024-03-04T16:28:11.057 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.057 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.058 DEBUG:boto:Params: {} 2024-03-04T16:28:11.058 DEBUG:boto:Token: None 2024-03-04T16:28:11.058 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E= 2024-03-04T16:28:11.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E=', 'Content-Length': '0'} 2024-03-04T16:28:11.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3be723c49846e7d-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.060 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.060 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.061 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.061 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.061 DEBUG:boto:Data: 2024-03-04T16:28:11.061 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.061 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.061 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.061 DEBUG:boto:Params: {} 2024-03-04T16:28:11.061 DEBUG:boto:Token: None 2024-03-04T16:28:11.061 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E= 2024-03-04T16:28:11.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E=', 'Content-Length': '0'} 2024-03-04T16:28:11.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9acfa2d7009096b-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:11.064 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.064 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.064 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.064 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.064 DEBUG:boto:Method: GET 2024-03-04T16:28:11.064 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.065 DEBUG:boto:Data: 2024-03-04T16:28:11.065 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.065 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.065 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.065 DEBUG:boto:Params: {} 2024-03-04T16:28:11.065 DEBUG:boto:Token: None 2024-03-04T16:28:11.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007afb6be1b4e39bb8-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.108 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.108 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.108 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.108 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.109 DEBUG:boto:Method: GET 2024-03-04T16:28:11.109 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.109 DEBUG:boto:Data: 2024-03-04T16:28:11.109 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.109 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.109 DEBUG:boto:Params: {} 2024-03-04T16:28:11.109 DEBUG:boto:Token: None 2024-03-04T16:28:11.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.109 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4744e190d98141a-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.112 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.112 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.112 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.113 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.113 DEBUG:boto:Method: GET 2024-03-04T16:28:11.113 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.113 DEBUG:boto:Data: 2024-03-04T16:28:11.113 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.113 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.113 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.114 DEBUG:boto:Params: {} 2024-03-04T16:28:11.114 DEBUG:boto:Token: None 2024-03-04T16:28:11.114 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.114 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005673efd6513f352-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.164 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.165 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.165 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.165 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.165 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.166 DEBUG:boto:Method: GET 2024-03-04T16:28:11.166 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.166 DEBUG:boto:Data: 2024-03-04T16:28:11.166 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.166 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.166 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.167 DEBUG:boto:Params: {} 2024-03-04T16:28:11.167 DEBUG:boto:Token: None 2024-03-04T16:28:11.167 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.167 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000439ba50c30752c0-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.170 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:11.171 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.171 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.171 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.171 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.171 DEBUG:boto:Method: GET 2024-03-04T16:28:11.171 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.171 DEBUG:boto:Data: 2024-03-04T16:28:11.171 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.171 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.171 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.171 DEBUG:boto:Params: {} 2024-03-04T16:28:11.172 DEBUG:boto:Token: None 2024-03-04T16:28:11.172 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.172 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000413e37f3b29f1067-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.175 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.175 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.175 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.175 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.175 DEBUG:boto:Method: GET 2024-03-04T16:28:11.175 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.175 DEBUG:boto:Data: 2024-03-04T16:28:11.175 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.175 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.176 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.176 DEBUG:boto:Params: {} 2024-03-04T16:28:11.177 DEBUG:boto:Token: None 2024-03-04T16:28:11.177 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.177 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b9a1c97572e2841-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.224 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.224 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.224 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.225 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.225 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.225 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.225 DEBUG:boto:Data: 2024-03-04T16:28:11.226 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.226 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.226 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.226 DEBUG:boto:Params: {} 2024-03-04T16:28:11.227 DEBUG:boto:Token: None 2024-03-04T16:28:11.227 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.227 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i4+nU1LySqjVjEZeaNdEbbv5O3Q= 2024-03-04T16:28:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i4+nU1LySqjVjEZeaNdEbbv5O3Q=', 'Content-Length': '0'} 2024-03-04T16:28:11.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009727f4eba3e3f6cf-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.230 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.230 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.230 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.231 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.231 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.231 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.232 DEBUG:boto:Data: 2024-03-04T16:28:11.232 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.232 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.232 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.233 DEBUG:boto:Params: {} 2024-03-04T16:28:11.233 DEBUG:boto:Token: None 2024-03-04T16:28:11.234 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i4+nU1LySqjVjEZeaNdEbbv5O3Q= 2024-03-04T16:28:11.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i4+nU1LySqjVjEZeaNdEbbv5O3Q=', 'Content-Length': '0'} 2024-03-04T16:28:11.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bd1da00ad3847f2-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:11.238 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.238 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.238 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.239 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.239 DEBUG:boto:Method: GET 2024-03-04T16:28:11.240 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.240 DEBUG:boto:Data: 2024-03-04T16:28:11.240 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.241 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.241 DEBUG:boto:Params: {} 2024-03-04T16:28:11.241 DEBUG:boto:Token: None 2024-03-04T16:28:11.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf0ffcbb090bae6-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.245 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.245 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.245 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.245 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.245 DEBUG:boto:Method: GET 2024-03-04T16:28:11.246 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.246 DEBUG:boto:Data: 2024-03-04T16:28:11.246 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.246 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.246 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.246 DEBUG:boto:Params: {} 2024-03-04T16:28:11.246 DEBUG:boto:Token: None 2024-03-04T16:28:11.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5b3136288b2bfb8-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.294 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.294 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.294 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.294 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.294 DEBUG:boto:Method: GET 2024-03-04T16:28:11.294 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.294 DEBUG:boto:Data: 2024-03-04T16:28:11.294 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.294 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.294 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.294 DEBUG:boto:Params: {} 2024-03-04T16:28:11.295 DEBUG:boto:Token: None 2024-03-04T16:28:11.295 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.295 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb4aaaa458eb54f2-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.301 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.301 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.301 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.301 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.301 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.301 DEBUG:boto:Method: GET 2024-03-04T16:28:11.302 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.302 DEBUG:boto:Data: 2024-03-04T16:28:11.302 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.302 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.302 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.302 DEBUG:boto:Params: {} 2024-03-04T16:28:11.302 DEBUG:boto:Token: None 2024-03-04T16:28:11.302 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec897875b864b739-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.352 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.353 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.353 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.353 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.353 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.353 DEBUG:boto:Data: 2024-03-04T16:28:11.353 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.353 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.353 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.353 DEBUG:boto:Params: {} 2024-03-04T16:28:11.354 DEBUG:boto:Token: None 2024-03-04T16:28:11.354 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2WmCjKI+LtTf1C+gCuhQkB3Md3c= 2024-03-04T16:28:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2WmCjKI+LtTf1C+gCuhQkB3Md3c=', 'Content-Length': '0'} 2024-03-04T16:28:11.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e363c72b49c64ece-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.356 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.356 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.356 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.357 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.357 DEBUG:boto:Data: 2024-03-04T16:28:11.357 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.357 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.357 DEBUG:boto:Params: {} 2024-03-04T16:28:11.357 DEBUG:boto:Token: None 2024-03-04T16:28:11.357 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2WmCjKI+LtTf1C+gCuhQkB3Md3c= 2024-03-04T16:28:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2WmCjKI+LtTf1C+gCuhQkB3Md3c=', 'Content-Length': '0'} 2024-03-04T16:28:11.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d8b9f223ccb69e6-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:11.360 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.360 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.361 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.361 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.361 DEBUG:boto:Method: GET 2024-03-04T16:28:11.361 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.361 DEBUG:boto:Data: 2024-03-04T16:28:11.361 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.361 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.361 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.361 DEBUG:boto:Params: {} 2024-03-04T16:28:11.361 DEBUG:boto:Token: None 2024-03-04T16:28:11.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000df2925da904777b-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.364 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.364 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:11.364 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.364 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.364 DEBUG:boto:Method: GET 2024-03-04T16:28:11.364 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.364 DEBUG:boto:Data: 2024-03-04T16:28:11.365 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.365 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.365 DEBUG:boto:Params: {} 2024-03-04T16:28:11.365 DEBUG:boto:Token: None 2024-03-04T16:28:11.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:11.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA= 2024-03-04T16:28:11.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY://10GqBm8Mu73+lqfTEusKzvAKA=', 'Content-Length': '0'} 2024-03-04T16:28:11.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097f33e0c6512bc84-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.412 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.412 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.412 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.412 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.413 DEBUG:boto:Method: GET 2024-03-04T16:28:11.413 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.413 DEBUG:boto:Data: 2024-03-04T16:28:11.413 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.413 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.413 DEBUG:boto:Params: {} 2024-03-04T16:28:11.413 DEBUG:boto:Token: None 2024-03-04T16:28:11.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.414 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8095f175e64924-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.417 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.417 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.418 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.418 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.418 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.418 DEBUG:boto:Method: GET 2024-03-04T16:28:11.418 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:11.418 DEBUG:boto:Data: 2024-03-04T16:28:11.418 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.418 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.418 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.418 DEBUG:boto:Params: {} 2024-03-04T16:28:11.419 DEBUG:boto:Token: None 2024-03-04T16:28:11.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc89e4682fc0a05a-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.421 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:11.422 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.422 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.423 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.423 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.424 DEBUG:boto:Method: GET 2024-03-04T16:28:11.424 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.424 DEBUG:boto:Data: 2024-03-04T16:28:11.424 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.424 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.424 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.424 DEBUG:boto:Params: {} 2024-03-04T16:28:11.425 DEBUG:boto:Token: None 2024-03-04T16:28:11.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1da3deb36c87e42-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.468 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.468 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.469 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.469 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.469 DEBUG:boto:Method: GET 2024-03-04T16:28:11.469 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.469 DEBUG:boto:Data: 2024-03-04T16:28:11.469 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.469 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.469 DEBUG:boto:Params: {} 2024-03-04T16:28:11.469 DEBUG:boto:Token: None 2024-03-04T16:28:11.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009caba7cba07e5137-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.513 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.513 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.514 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.514 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.514 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.514 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.514 DEBUG:boto:Data: 2024-03-04T16:28:11.514 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.514 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.514 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.514 DEBUG:boto:Params: {} 2024-03-04T16:28:11.515 DEBUG:boto:Token: None 2024-03-04T16:28:11.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JcloHJbJBUGcXLtiXikxUZ2FVQE= 2024-03-04T16:28:11.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JcloHJbJBUGcXLtiXikxUZ2FVQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82188daa2bd0e7d-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.517 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.518 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.518 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.518 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.518 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.518 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.518 DEBUG:boto:Data: 2024-03-04T16:28:11.519 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.519 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.519 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.519 DEBUG:boto:Params: {} 2024-03-04T16:28:11.519 DEBUG:boto:Token: None 2024-03-04T16:28:11.520 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JcloHJbJBUGcXLtiXikxUZ2FVQE= 2024-03-04T16:28:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JcloHJbJBUGcXLtiXikxUZ2FVQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fade60524f95e2cf-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:11.523 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.523 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.523 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.523 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.524 DEBUG:boto:Method: GET 2024-03-04T16:28:11.524 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.524 DEBUG:boto:Data: 2024-03-04T16:28:11.524 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.525 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.525 DEBUG:boto:Params: {} 2024-03-04T16:28:11.525 DEBUG:boto:Token: None 2024-03-04T16:28:11.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002735dbc86509ccf9-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.568 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.568 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.568 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.569 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.569 DEBUG:boto:Method: GET 2024-03-04T16:28:11.569 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.569 DEBUG:boto:Data: 2024-03-04T16:28:11.569 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.569 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.569 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.569 DEBUG:boto:Params: {} 2024-03-04T16:28:11.570 DEBUG:boto:Token: None 2024-03-04T16:28:11.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b38cc56befba1a94-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.612 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.612 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.612 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.612 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.613 DEBUG:boto:Method: GET 2024-03-04T16:28:11.613 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.613 DEBUG:boto:Data: 2024-03-04T16:28:11.613 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.613 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.613 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.613 DEBUG:boto:Params: {} 2024-03-04T16:28:11.613 DEBUG:boto:Token: None 2024-03-04T16:28:11.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d928bff8a159d8-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.616 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.617 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.617 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.617 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.617 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.617 DEBUG:boto:Method: GET 2024-03-04T16:28:11.617 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.618 DEBUG:boto:Data: 2024-03-04T16:28:11.618 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.618 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.618 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.618 DEBUG:boto:Params: {} 2024-03-04T16:28:11.618 DEBUG:boto:Token: None 2024-03-04T16:28:11.618 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaa737e665f3d85d-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.621 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.621 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.621 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.622 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.622 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.622 DEBUG:boto:Data: 2024-03-04T16:28:11.622 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.622 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.622 DEBUG:boto:Params: {} 2024-03-04T16:28:11.622 DEBUG:boto:Token: None 2024-03-04T16:28:11.622 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y4OGP7BZfYD55+XRfxJX0R8OAFk= 2024-03-04T16:28:11.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y4OGP7BZfYD55+XRfxJX0R8OAFk=', 'Content-Length': '0'} 2024-03-04T16:28:11.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af8e3b89079711dd-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.625 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.625 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.625 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.626 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.626 DEBUG:boto:Data: 2024-03-04T16:28:11.626 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.626 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.626 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.626 DEBUG:boto:Params: {} 2024-03-04T16:28:11.626 DEBUG:boto:Token: None 2024-03-04T16:28:11.626 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y4OGP7BZfYD55+XRfxJX0R8OAFk= 2024-03-04T16:28:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y4OGP7BZfYD55+XRfxJX0R8OAFk=', 'Content-Length': '0'} 2024-03-04T16:28:11.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8ce02af36ef0dec-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:11.629 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.629 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.629 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.629 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.630 DEBUG:boto:Method: GET 2024-03-04T16:28:11.630 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.630 DEBUG:boto:Data: 2024-03-04T16:28:11.630 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.630 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.630 DEBUG:boto:Params: {} 2024-03-04T16:28:11.630 DEBUG:boto:Token: None 2024-03-04T16:28:11.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000901b625eab432a2c-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.676 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.676 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:11.676 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.676 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.677 DEBUG:boto:Method: GET 2024-03-04T16:28:11.677 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.677 DEBUG:boto:Data: 2024-03-04T16:28:11.677 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.677 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.677 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.677 DEBUG:boto:Params: {} 2024-03-04T16:28:11.677 DEBUG:boto:Token: None 2024-03-04T16:28:11.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:11.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek= 2024-03-04T16:28:11.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7BMcJqFp8LpgvvsRiODdZUzfgek=', 'Content-Length': '0'} 2024-03-04T16:28:11.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a71e2421bb364eca-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.720 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.720 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.720 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.720 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.721 DEBUG:boto:Method: GET 2024-03-04T16:28:11.721 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.721 DEBUG:boto:Data: 2024-03-04T16:28:11.721 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:11.721 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.721 DEBUG:boto:Params: {} 2024-03-04T16:28:11.721 DEBUG:boto:Token: None 2024-03-04T16:28:11.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4392d2fdadcf14-0065e5f69b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.768 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.769 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.769 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.769 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.769 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.769 DEBUG:boto:Method: GET 2024-03-04T16:28:11.769 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:11.769 DEBUG:boto:Data: 2024-03-04T16:28:11.769 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.770 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.770 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.770 DEBUG:boto:Params: {} 2024-03-04T16:28:11.770 DEBUG:boto:Token: None 2024-03-04T16:28:11.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.770 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ea7057294133666-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.773 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone2, test-zone1} 2024-03-04T16:28:11.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone3, test-zone1} 2024-03-04T16:28:11.774 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.774 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.774 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.774 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:11.775 DEBUG:boto:Data: 2024-03-04T16:28:11.775 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.775 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.775 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.775 DEBUG:boto:Params: {} 2024-03-04T16:28:11.775 DEBUG:boto:Token: None 2024-03-04T16:28:11.775 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.775 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JphoalLWNwtBfWW8OFDHJw1J8pA= 2024-03-04T16:28:11.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JphoalLWNwtBfWW8OFDHJw1J8pA=', 'Content-Length': '0'} 2024-03-04T16:28:11.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d173c55ec982468-0065e5f69b-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.778 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.778 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.779 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.779 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:11.779 DEBUG:boto:Data: 2024-03-04T16:28:11.779 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.779 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.779 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.780 DEBUG:boto:Params: {} 2024-03-04T16:28:11.780 DEBUG:boto:Token: None 2024-03-04T16:28:11.780 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.780 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JphoalLWNwtBfWW8OFDHJw1J8pA= 2024-03-04T16:28:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JphoalLWNwtBfWW8OFDHJw1J8pA=', 'Content-Length': '0'} 2024-03-04T16:28:11.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009082f49b4908a739-0065e5f69b-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.788 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:11.788 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.788 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.788 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.789 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.789 DEBUG:boto:Method: GET 2024-03-04T16:28:11.789 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.789 DEBUG:boto:Data: 2024-03-04T16:28:11.789 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.789 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.789 DEBUG:boto:Params: {} 2024-03-04T16:28:11.789 DEBUG:boto:Token: None 2024-03-04T16:28:11.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/?versions 2024-03-04T16:28:11.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE= 2024-03-04T16:28:11.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c96063988981ba-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.793 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.794 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:11.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:11.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:11.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:11.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:11.795 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.795 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.795 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.795 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.795 DEBUG:boto:Method: GET 2024-03-04T16:28:11.795 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.795 DEBUG:boto:Data: 2024-03-04T16:28:11.795 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.795 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.795 DEBUG:boto:Params: {} 2024-03-04T16:28:11.796 DEBUG:boto:Token: None 2024-03-04T16:28:11.796 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/?versions 2024-03-04T16:28:11.796 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE= 2024-03-04T16:28:11.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000456c809a972d89f7-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.799 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.800 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:11.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:11.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:11.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:11.800 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.800 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.801 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.801 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.801 DEBUG:boto:Method: GET 2024-03-04T16:28:11.801 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.801 DEBUG:boto:Data: 2024-03-04T16:28:11.801 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.801 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.801 DEBUG:boto:Params: {} 2024-03-04T16:28:11.802 DEBUG:boto:Token: None 2024-03-04T16:28:11.802 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/?versions 2024-03-04T16:28:11.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE= 2024-03-04T16:28:11.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004361ced0871bc57b-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.806 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.807 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.807 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.807 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.807 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.807 DEBUG:boto:Method: GET 2024-03-04T16:28:11.807 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:11.807 DEBUG:boto:Data: 2024-03-04T16:28:11.807 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.807 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.808 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.810 DEBUG:boto:Params: {} 2024-03-04T16:28:11.811 DEBUG:boto:Token: None 2024-03-04T16:28:11.812 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/?versions 2024-03-04T16:28:11.812 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE= 2024-03-04T16:28:11.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1S8LnsULIRDCe+JcQeVqVacujQE=', 'Content-Length': '0'} 2024-03-04T16:28:11.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060dd551042837239-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.815 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:11.816 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.816 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.816 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.816 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.816 DEBUG:boto:Method: GET 2024-03-04T16:28:11.816 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.817 DEBUG:boto:Data: 2024-03-04T16:28:11.817 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.817 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.817 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.817 DEBUG:boto:Params: {} 2024-03-04T16:28:11.817 DEBUG:boto:Token: None 2024-03-04T16:28:11.817 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.817 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052082ffa66ec68a6-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.820 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.820 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.820 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.820 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.820 DEBUG:boto:Method: GET 2024-03-04T16:28:11.820 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.820 DEBUG:boto:Data: 2024-03-04T16:28:11.820 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.821 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.821 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.821 DEBUG:boto:Params: {} 2024-03-04T16:28:11.821 DEBUG:boto:Token: None 2024-03-04T16:28:11.821 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.821 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006283612eb4b6903b-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.825 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.825 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.825 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.825 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.825 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.825 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.825 DEBUG:boto:Data: 2024-03-04T16:28:11.825 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.826 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.826 DEBUG:boto:Params: {} 2024-03-04T16:28:11.826 DEBUG:boto:Token: None 2024-03-04T16:28:11.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.826 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpWslf6Gt263M7xGGKMJkLsSMXM= 2024-03-04T16:28:11.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpWslf6Gt263M7xGGKMJkLsSMXM=', 'Content-Length': '0'} 2024-03-04T16:28:11.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042abd75eb4e2e7cc-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.828 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.828 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.829 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.829 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.829 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.829 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.829 DEBUG:boto:Data: 2024-03-04T16:28:11.829 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.829 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.829 DEBUG:boto:Params: {} 2024-03-04T16:28:11.829 DEBUG:boto:Token: None 2024-03-04T16:28:11.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rpWslf6Gt263M7xGGKMJkLsSMXM= 2024-03-04T16:28:11.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rpWslf6Gt263M7xGGKMJkLsSMXM=', 'Content-Length': '0'} 2024-03-04T16:28:11.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000387d43176d8b2888-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:11.832 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.832 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.832 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.832 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.832 DEBUG:boto:Method: GET 2024-03-04T16:28:11.832 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.832 DEBUG:boto:Data: 2024-03-04T16:28:11.833 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.833 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.833 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.833 DEBUG:boto:Params: {} 2024-03-04T16:28:11.833 DEBUG:boto:Token: None 2024-03-04T16:28:11.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df92acf32eea0eb4-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.835 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.835 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.835 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.835 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.835 DEBUG:boto:Method: GET 2024-03-04T16:28:11.836 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.836 DEBUG:boto:Data: 2024-03-04T16:28:11.836 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.836 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.836 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.836 DEBUG:boto:Params: {} 2024-03-04T16:28:11.836 DEBUG:boto:Token: None 2024-03-04T16:28:11.836 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.836 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d56cb0c54fe394ff-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.839 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.839 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.839 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.839 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.839 DEBUG:boto:Method: GET 2024-03-04T16:28:11.839 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.839 DEBUG:boto:Data: 2024-03-04T16:28:11.839 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.839 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.839 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.840 DEBUG:boto:Params: {} 2024-03-04T16:28:11.840 DEBUG:boto:Token: None 2024-03-04T16:28:11.840 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.840 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc4bbccdafba646-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.843 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.843 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.843 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.843 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.843 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.844 DEBUG:boto:Method: GET 2024-03-04T16:28:11.844 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.844 DEBUG:boto:Data: 2024-03-04T16:28:11.844 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.844 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.844 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.844 DEBUG:boto:Params: {} 2024-03-04T16:28:11.844 DEBUG:boto:Token: None 2024-03-04T16:28:11.844 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004197c75920e3749a-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.847 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.847 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.848 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.848 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.848 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:11.848 DEBUG:boto:Data: 2024-03-04T16:28:11.848 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.848 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.848 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.848 DEBUG:boto:Params: {} 2024-03-04T16:28:11.849 DEBUG:boto:Token: None 2024-03-04T16:28:11.849 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy 2024-03-04T16:28:11.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2dW3rIGHGzLMyD8gg0BG8WV/do= 2024-03-04T16:28:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2dW3rIGHGzLMyD8gg0BG8WV/do=', 'Content-Length': '0'} 2024-03-04T16:28:11.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d9e2bc3201f280c-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.851 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.851 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.851 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.852 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:11.852 DEBUG:boto:Data: 2024-03-04T16:28:11.852 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.852 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.852 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.852 DEBUG:boto:Params: {} 2024-03-04T16:28:11.852 DEBUG:boto:Token: None 2024-03-04T16:28:11.852 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy 2024-03-04T16:28:11.852 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2dW3rIGHGzLMyD8gg0BG8WV/do= 2024-03-04T16:28:11.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2dW3rIGHGzLMyD8gg0BG8WV/do=', 'Content-Length': '0'} 2024-03-04T16:28:11.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c435fbc0d771a0e4-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:11.856 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.856 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.856 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.856 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.856 DEBUG:boto:Method: GET 2024-03-04T16:28:11.856 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.856 DEBUG:boto:Data: 2024-03-04T16:28:11.856 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.856 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.857 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.857 DEBUG:boto:Params: {} 2024-03-04T16:28:11.857 DEBUG:boto:Token: None 2024-03-04T16:28:11.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000030ed564797c6156-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.859 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:11.859 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:11.859 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.860 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.860 DEBUG:boto:Method: GET 2024-03-04T16:28:11.860 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.860 DEBUG:boto:Data: 2024-03-04T16:28:11.860 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.860 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.860 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.860 DEBUG:boto:Params: {} 2024-03-04T16:28:11.860 DEBUG:boto:Token: None 2024-03-04T16:28:11.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:11.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw= 2024-03-04T16:28:11.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Odzxwvi9kRNc4QFcgyF2qjX/ULw=', 'Content-Length': '0'} 2024-03-04T16:28:11.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000672ecfb3c7d29677-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.863 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.863 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.863 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.863 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.863 DEBUG:boto:Method: GET 2024-03-04T16:28:11.863 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.863 DEBUG:boto:Data: 2024-03-04T16:28:11.864 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.864 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.864 DEBUG:boto:Params: {} 2024-03-04T16:28:11.864 DEBUG:boto:Token: None 2024-03-04T16:28:11.864 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.864 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd41b2a5d3854bc-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.868 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.868 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.868 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.869 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.869 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.869 DEBUG:boto:Method: GET 2024-03-04T16:28:11.869 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:11.869 DEBUG:boto:Data: 2024-03-04T16:28:11.869 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.869 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.869 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.870 DEBUG:boto:Params: {} 2024-03-04T16:28:11.870 DEBUG:boto:Token: None 2024-03-04T16:28:11.870 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.870 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078703130399bcd4d-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.872 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:11.873 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.873 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.873 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.873 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.873 DEBUG:boto:Method: GET 2024-03-04T16:28:11.874 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.874 DEBUG:boto:Data: 2024-03-04T16:28:11.874 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.874 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.874 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.874 DEBUG:boto:Params: {} 2024-03-04T16:28:11.874 DEBUG:boto:Token: None 2024-03-04T16:28:11.874 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.875 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b80d78fa18dbff5-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.877 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.877 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.877 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.878 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.878 DEBUG:boto:Method: GET 2024-03-04T16:28:11.878 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.878 DEBUG:boto:Data: 2024-03-04T16:28:11.878 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.879 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.879 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.879 DEBUG:boto:Params: {} 2024-03-04T16:28:11.879 DEBUG:boto:Token: None 2024-03-04T16:28:11.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e09c059c43fc2b9e-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.924 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.925 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.925 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.925 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.925 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.925 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.925 DEBUG:boto:Data: 2024-03-04T16:28:11.925 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.925 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.925 DEBUG:boto:Params: {} 2024-03-04T16:28:11.926 DEBUG:boto:Token: None 2024-03-04T16:28:11.926 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DUvOHfB9dojpTJROvKn15aEPLgs= 2024-03-04T16:28:11.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DUvOHfB9dojpTJROvKn15aEPLgs=', 'Content-Length': '0'} 2024-03-04T16:28:11.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074d4c306a54e7375-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.928 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.928 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.928 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.929 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.929 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.929 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.929 DEBUG:boto:Data: 2024-03-04T16:28:11.929 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.929 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.929 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.929 DEBUG:boto:Params: {} 2024-03-04T16:28:11.929 DEBUG:boto:Token: None 2024-03-04T16:28:11.929 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DUvOHfB9dojpTJROvKn15aEPLgs= 2024-03-04T16:28:11.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DUvOHfB9dojpTJROvKn15aEPLgs=', 'Content-Length': '0'} 2024-03-04T16:28:11.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b11d00bd09d596b-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:11.932 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.932 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.932 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.932 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.933 DEBUG:boto:Method: GET 2024-03-04T16:28:11.933 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.933 DEBUG:boto:Data: 2024-03-04T16:28:11.933 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.933 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.933 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.933 DEBUG:boto:Params: {} 2024-03-04T16:28:11.933 DEBUG:boto:Token: None 2024-03-04T16:28:11.933 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.933 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0b5903b7d4decbd-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.976 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.977 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.977 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.977 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.977 DEBUG:boto:Method: GET 2024-03-04T16:28:11.977 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.977 DEBUG:boto:Data: 2024-03-04T16:28:11.977 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.977 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.977 DEBUG:boto:Params: {} 2024-03-04T16:28:11.978 DEBUG:boto:Token: None 2024-03-04T16:28:11.978 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:11.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086903a88b3a71a86-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.982 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.982 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.982 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.982 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.982 DEBUG:boto:Method: GET 2024-03-04T16:28:11.982 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.982 DEBUG:boto:Data: 2024-03-04T16:28:11.982 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.982 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.982 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.983 DEBUG:boto:Params: {} 2024-03-04T16:28:11.983 DEBUG:boto:Token: None 2024-03-04T16:28:11.983 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.983 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a3befed41325aa-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.986 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.986 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:11.986 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:11.986 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.986 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.986 DEBUG:boto:Method: GET 2024-03-04T16:28:11.986 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:11.987 DEBUG:boto:Data: 2024-03-04T16:28:11.987 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.987 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.987 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.987 DEBUG:boto:Params: {} 2024-03-04T16:28:11.987 DEBUG:boto:Token: None 2024-03-04T16:28:11.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/ 2024-03-04T16:28:11.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8= 2024-03-04T16:28:11.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oh5BQyL2VUL6NLSPqI0nJTR5vg8=', 'Content-Length': '0'} 2024-03-04T16:28:11.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000175caece2e3e8cce-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.990 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:11.990 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.990 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.991 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.991 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.991 DEBUG:boto:Data: 2024-03-04T16:28:11.991 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.991 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.991 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.991 DEBUG:boto:Params: {} 2024-03-04T16:28:11.991 DEBUG:boto:Token: None 2024-03-04T16:28:11.991 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.991 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E= 2024-03-04T16:28:11.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E=', 'Content-Length': '0'} 2024-03-04T16:28:11.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073dc5bc740ab5d41-0065e5f69b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.994 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.994 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.995 DEBUG:boto:Method: HEAD 2024-03-04T16:28:11.995 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.995 DEBUG:boto:Data: 2024-03-04T16:28:11.995 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.995 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:11.995 DEBUG:boto:Port: 8000 2024-03-04T16:28:11.995 DEBUG:boto:Params: {} 2024-03-04T16:28:11.995 DEBUG:boto:Token: None 2024-03-04T16:28:11.995 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.996 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E= 2024-03-04T16:28:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nKqE0dWVkig53Iv7h2Gap/bkC2E=', 'Content-Length': '0'} 2024-03-04T16:28:11.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8f5fb60d6c017f3-0065e5f69b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:11 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:11.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:11.998 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.998 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:11.998 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.998 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.998 DEBUG:boto:Method: GET 2024-03-04T16:28:11.999 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.999 DEBUG:boto:Data: 2024-03-04T16:28:11.999 DEBUG:boto:Headers: {} 2024-03-04T16:28:11.999 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:11.999 DEBUG:boto:Port: 8001 2024-03-04T16:28:11.999 DEBUG:boto:Params: {} 2024-03-04T16:28:11.999 DEBUG:boto:Token: None 2024-03-04T16:28:11.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:11 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:11.999 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg= 2024-03-04T16:28:11.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:11 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:teWyTwGNU1HZoL2ymGh6YZal0vg=', 'Content-Length': '0'} 2024-03-04T16:28:12.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082749d39e22c3079-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.044 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.044 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.044 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.044 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.044 DEBUG:boto:Method: GET 2024-03-04T16:28:12.045 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.045 DEBUG:boto:Data: 2024-03-04T16:28:12.045 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.045 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.045 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.045 DEBUG:boto:Params: {} 2024-03-04T16:28:12.045 DEBUG:boto:Token: None 2024-03-04T16:28:12.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.045 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008a3dbb0d8c91d07-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.048 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.048 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.048 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.048 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.048 DEBUG:boto:Method: GET 2024-03-04T16:28:12.048 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.049 DEBUG:boto:Data: 2024-03-04T16:28:12.049 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.049 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.049 DEBUG:boto:Params: {} 2024-03-04T16:28:12.049 DEBUG:boto:Token: None 2024-03-04T16:28:12.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c42292f06356068-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.096 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.097 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.097 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.097 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.097 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.097 DEBUG:boto:Method: GET 2024-03-04T16:28:12.097 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.097 DEBUG:boto:Data: 2024-03-04T16:28:12.097 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.097 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.097 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.097 DEBUG:boto:Params: {} 2024-03-04T16:28:12.098 DEBUG:boto:Token: None 2024-03-04T16:28:12.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.098 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccae9ae483d4e21-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.101 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:12.102 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.102 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.102 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.102 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.102 DEBUG:boto:Method: GET 2024-03-04T16:28:12.102 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.102 DEBUG:boto:Data: 2024-03-04T16:28:12.102 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.102 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.102 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.102 DEBUG:boto:Params: {} 2024-03-04T16:28:12.103 DEBUG:boto:Token: None 2024-03-04T16:28:12.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dc52c9f20d5fbcc-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.152 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.152 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.152 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.152 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.153 DEBUG:boto:Method: GET 2024-03-04T16:28:12.153 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.153 DEBUG:boto:Data: 2024-03-04T16:28:12.153 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.153 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.153 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.153 DEBUG:boto:Params: {} 2024-03-04T16:28:12.153 DEBUG:boto:Token: None 2024-03-04T16:28:12.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008784d1502385168-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.197 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.197 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.197 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.197 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.197 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.197 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.197 DEBUG:boto:Data: 2024-03-04T16:28:12.197 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.198 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.198 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.198 DEBUG:boto:Params: {} 2024-03-04T16:28:12.198 DEBUG:boto:Token: None 2024-03-04T16:28:12.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sIMi4CJLDjFiiCQ7KyuZnZKoMCk= 2024-03-04T16:28:12.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sIMi4CJLDjFiiCQ7KyuZnZKoMCk=', 'Content-Length': '0'} 2024-03-04T16:28:12.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb39ed1cc3172e9e-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.202 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.202 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.202 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.202 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.203 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.203 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.203 DEBUG:boto:Data: 2024-03-04T16:28:12.203 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.203 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.203 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.203 DEBUG:boto:Params: {} 2024-03-04T16:28:12.203 DEBUG:boto:Token: None 2024-03-04T16:28:12.203 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.204 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sIMi4CJLDjFiiCQ7KyuZnZKoMCk= 2024-03-04T16:28:12.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sIMi4CJLDjFiiCQ7KyuZnZKoMCk=', 'Content-Length': '0'} 2024-03-04T16:28:12.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bba1105027eab66b-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:12.208 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.209 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.209 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.209 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.209 DEBUG:boto:Method: GET 2024-03-04T16:28:12.210 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.210 DEBUG:boto:Data: 2024-03-04T16:28:12.210 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.210 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.210 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.210 DEBUG:boto:Params: {} 2024-03-04T16:28:12.210 DEBUG:boto:Token: None 2024-03-04T16:28:12.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.211 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6b95c702be512a3-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.256 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.256 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.257 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.257 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.257 DEBUG:boto:Method: GET 2024-03-04T16:28:12.257 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.257 DEBUG:boto:Data: 2024-03-04T16:28:12.257 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.258 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.258 DEBUG:boto:Params: {} 2024-03-04T16:28:12.258 DEBUG:boto:Token: None 2024-03-04T16:28:12.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e748b4db0faf880-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.304 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.305 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.305 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.305 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.305 DEBUG:boto:Method: GET 2024-03-04T16:28:12.305 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.305 DEBUG:boto:Data: 2024-03-04T16:28:12.305 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.305 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.307 DEBUG:boto:Params: {} 2024-03-04T16:28:12.307 DEBUG:boto:Token: None 2024-03-04T16:28:12.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb4001e081813d2-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.310 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.311 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.311 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.311 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.311 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.312 DEBUG:boto:Method: GET 2024-03-04T16:28:12.312 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.312 DEBUG:boto:Data: 2024-03-04T16:28:12.312 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.312 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.312 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.312 DEBUG:boto:Params: {} 2024-03-04T16:28:12.312 DEBUG:boto:Token: None 2024-03-04T16:28:12.312 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.313 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1d584cc2aec513-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.316 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.316 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.316 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.316 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.317 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.317 DEBUG:boto:Data: 2024-03-04T16:28:12.317 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.317 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.317 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.317 DEBUG:boto:Params: {} 2024-03-04T16:28:12.317 DEBUG:boto:Token: None 2024-03-04T16:28:12.317 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.317 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Czu/Tu6L0+i8DtzShVx7QOGXc6g= 2024-03-04T16:28:12.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Czu/Tu6L0+i8DtzShVx7QOGXc6g=', 'Content-Length': '0'} 2024-03-04T16:28:12.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002faef8f0d1f7fcdf-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.320 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.320 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.320 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.320 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.321 DEBUG:boto:Data: 2024-03-04T16:28:12.321 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.321 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.321 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.321 DEBUG:boto:Params: {} 2024-03-04T16:28:12.321 DEBUG:boto:Token: None 2024-03-04T16:28:12.321 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.321 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Czu/Tu6L0+i8DtzShVx7QOGXc6g= 2024-03-04T16:28:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Czu/Tu6L0+i8DtzShVx7QOGXc6g=', 'Content-Length': '0'} 2024-03-04T16:28:12.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ad72e5f5eefa9e1-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:12.324 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.324 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.324 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.324 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.324 DEBUG:boto:Method: GET 2024-03-04T16:28:12.324 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.324 DEBUG:boto:Data: 2024-03-04T16:28:12.325 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.325 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.325 DEBUG:boto:Params: {} 2024-03-04T16:28:12.325 DEBUG:boto:Token: None 2024-03-04T16:28:12.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.325 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccb4bfcaaa67727f-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.373 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.374 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:12.374 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.374 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.374 DEBUG:boto:Method: GET 2024-03-04T16:28:12.375 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.375 DEBUG:boto:Data: 2024-03-04T16:28:12.375 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.375 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.375 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.375 DEBUG:boto:Params: {} 2024-03-04T16:28:12.375 DEBUG:boto:Token: None 2024-03-04T16:28:12.375 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:12.376 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg= 2024-03-04T16:28:12.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:glGlqsIhdqM7jjO7WEJaLCHqtSg=', 'Content-Length': '0'} 2024-03-04T16:28:12.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ae1e37bbcb5479a-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.420 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.420 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.420 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.420 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.421 DEBUG:boto:Method: GET 2024-03-04T16:28:12.421 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.421 DEBUG:boto:Data: 2024-03-04T16:28:12.421 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.421 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.421 DEBUG:boto:Params: {} 2024-03-04T16:28:12.421 DEBUG:boto:Token: None 2024-03-04T16:28:12.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086521b84d7d3171e-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.424 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.425 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.425 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.425 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.425 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.425 DEBUG:boto:Method: GET 2024-03-04T16:28:12.425 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:12.426 DEBUG:boto:Data: 2024-03-04T16:28:12.426 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.427 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.427 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.427 DEBUG:boto:Params: {} 2024-03-04T16:28:12.427 DEBUG:boto:Token: None 2024-03-04T16:28:12.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.427 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000887faee4cb69d31a-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.430 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:12.431 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.431 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.431 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.431 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.431 DEBUG:boto:Method: GET 2024-03-04T16:28:12.431 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.431 DEBUG:boto:Data: 2024-03-04T16:28:12.432 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.432 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.432 DEBUG:boto:Params: {} 2024-03-04T16:28:12.432 DEBUG:boto:Token: None 2024-03-04T16:28:12.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008232f764b75641a4-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.435 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.435 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.435 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.435 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.435 DEBUG:boto:Method: GET 2024-03-04T16:28:12.435 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.435 DEBUG:boto:Data: 2024-03-04T16:28:12.435 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.436 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.436 DEBUG:boto:Params: {} 2024-03-04T16:28:12.436 DEBUG:boto:Token: None 2024-03-04T16:28:12.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.436 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bdbf843fda2eaa8-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.480 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.480 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.480 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.480 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.481 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.481 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.481 DEBUG:boto:Data: 2024-03-04T16:28:12.481 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.481 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.481 DEBUG:boto:Params: {} 2024-03-04T16:28:12.481 DEBUG:boto:Token: None 2024-03-04T16:28:12.481 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XNWkwI/WNmSZTBDkr0LuCIJtmY0= 2024-03-04T16:28:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XNWkwI/WNmSZTBDkr0LuCIJtmY0=', 'Content-Length': '0'} 2024-03-04T16:28:12.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d0df2c18942967a-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.484 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.484 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.484 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.485 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.485 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.485 DEBUG:boto:Data: 2024-03-04T16:28:12.485 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.485 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.485 DEBUG:boto:Params: {} 2024-03-04T16:28:12.485 DEBUG:boto:Token: None 2024-03-04T16:28:12.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XNWkwI/WNmSZTBDkr0LuCIJtmY0= 2024-03-04T16:28:12.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XNWkwI/WNmSZTBDkr0LuCIJtmY0=', 'Content-Length': '0'} 2024-03-04T16:28:12.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cfec0197df25c53-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:12.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.488 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.488 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.489 DEBUG:boto:Method: GET 2024-03-04T16:28:12.489 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.489 DEBUG:boto:Data: 2024-03-04T16:28:12.489 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.489 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.489 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.489 DEBUG:boto:Params: {} 2024-03-04T16:28:12.489 DEBUG:boto:Token: None 2024-03-04T16:28:12.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.490 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036e9d65c0849c637-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.492 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.492 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.492 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.492 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.492 DEBUG:boto:Method: GET 2024-03-04T16:28:12.492 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.492 DEBUG:boto:Data: 2024-03-04T16:28:12.492 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.493 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.493 DEBUG:boto:Params: {} 2024-03-04T16:28:12.493 DEBUG:boto:Token: None 2024-03-04T16:28:12.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042acefb0c4a5e581-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.536 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.536 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.536 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.536 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.537 DEBUG:boto:Method: GET 2024-03-04T16:28:12.537 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.537 DEBUG:boto:Data: 2024-03-04T16:28:12.537 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.537 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.537 DEBUG:boto:Params: {} 2024-03-04T16:28:12.537 DEBUG:boto:Token: None 2024-03-04T16:28:12.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c299ec8906d9a2-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.541 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.541 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.541 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.541 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.542 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.542 DEBUG:boto:Method: GET 2024-03-04T16:28:12.542 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.542 DEBUG:boto:Data: 2024-03-04T16:28:12.542 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.542 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.542 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.542 DEBUG:boto:Params: {} 2024-03-04T16:28:12.542 DEBUG:boto:Token: None 2024-03-04T16:28:12.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000299f73099ff5cb3c-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.546 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.546 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.547 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.547 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.547 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.547 DEBUG:boto:Data: 2024-03-04T16:28:12.547 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.547 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.547 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.547 DEBUG:boto:Params: {} 2024-03-04T16:28:12.547 DEBUG:boto:Token: None 2024-03-04T16:28:12.548 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1tml6zQ3sidra6Ivr/oZYYkNXY= 2024-03-04T16:28:12.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1tml6zQ3sidra6Ivr/oZYYkNXY=', 'Content-Length': '0'} 2024-03-04T16:28:12.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfc552f4e8802bb7-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.550 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.551 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.551 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.551 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.552 DEBUG:boto:Data: 2024-03-04T16:28:12.552 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.552 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.552 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.552 DEBUG:boto:Params: {} 2024-03-04T16:28:12.552 DEBUG:boto:Token: None 2024-03-04T16:28:12.552 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x1tml6zQ3sidra6Ivr/oZYYkNXY= 2024-03-04T16:28:12.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x1tml6zQ3sidra6Ivr/oZYYkNXY=', 'Content-Length': '0'} 2024-03-04T16:28:12.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056396ca5cf33b906-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:12.556 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.556 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.556 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.556 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.557 DEBUG:boto:Method: GET 2024-03-04T16:28:12.557 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.557 DEBUG:boto:Data: 2024-03-04T16:28:12.557 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.558 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.558 DEBUG:boto:Params: {} 2024-03-04T16:28:12.558 DEBUG:boto:Token: None 2024-03-04T16:28:12.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5c0036d8b4571e4-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.604 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.604 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:12.604 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.604 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.605 DEBUG:boto:Method: GET 2024-03-04T16:28:12.605 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.605 DEBUG:boto:Data: 2024-03-04T16:28:12.605 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.605 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.605 DEBUG:boto:Params: {} 2024-03-04T16:28:12.605 DEBUG:boto:Token: None 2024-03-04T16:28:12.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:12.606 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0= 2024-03-04T16:28:12.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Yh6vs/Er6mlPM9q6HvoD7ji1T0=', 'Content-Length': '0'} 2024-03-04T16:28:12.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008daca1a04a9e36c2-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.648 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.648 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.648 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.648 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.649 DEBUG:boto:Method: GET 2024-03-04T16:28:12.649 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.649 DEBUG:boto:Data: 2024-03-04T16:28:12.649 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.649 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.649 DEBUG:boto:Params: {} 2024-03-04T16:28:12.649 DEBUG:boto:Token: None 2024-03-04T16:28:12.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a55aec8b53726d4-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.652 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.653 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.653 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.653 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.653 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.653 DEBUG:boto:Method: GET 2024-03-04T16:28:12.653 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:12.653 DEBUG:boto:Data: 2024-03-04T16:28:12.653 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:12.654 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.654 DEBUG:boto:Params: {} 2024-03-04T16:28:12.654 DEBUG:boto:Token: None 2024-03-04T16:28:12.654 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa36a8974c161515-0065e5f69c-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.657 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone3, test-zone1} 2024-03-04T16:28:12.658 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-21 zones={test-zone3, test-zone2} 2024-03-04T16:28:12.658 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.658 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.658 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.658 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:12.658 DEBUG:boto:Data: 2024-03-04T16:28:12.658 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.660 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.660 DEBUG:boto:Params: {} 2024-03-04T16:28:12.660 DEBUG:boto:Token: None 2024-03-04T16:28:12.660 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s9FR3lOjwDhS62NYpffVIDd0aTY= 2024-03-04T16:28:12.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s9FR3lOjwDhS62NYpffVIDd0aTY=', 'Content-Length': '0'} 2024-03-04T16:28:12.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de9f3990d47f4bc1-0065e5f69c-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.663 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.663 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.663 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.664 DEBUG:boto:Path: /sbnvju-21/ 2024-03-04T16:28:12.664 DEBUG:boto:Data: 2024-03-04T16:28:12.664 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.664 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.664 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.664 DEBUG:boto:Params: {} 2024-03-04T16:28:12.664 DEBUG:boto:Token: None 2024-03-04T16:28:12.664 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.664 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s9FR3lOjwDhS62NYpffVIDd0aTY= 2024-03-04T16:28:12.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s9FR3lOjwDhS62NYpffVIDd0aTY=', 'Content-Length': '0'} 2024-03-04T16:28:12.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b98949e46282150d-0065e5f69c-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:28:12.667 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.667 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.667 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.667 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.667 DEBUG:boto:Method: GET 2024-03-04T16:28:12.668 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.668 DEBUG:boto:Data: 2024-03-04T16:28:12.668 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.668 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.668 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.668 DEBUG:boto:Params: {} 2024-03-04T16:28:12.668 DEBUG:boto:Token: None 2024-03-04T16:28:12.668 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/?versions 2024-03-04T16:28:12.668 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08= 2024-03-04T16:28:12.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08=', 'Content-Length': '0'} 2024-03-04T16:28:12.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009218a7ad58f0a20d-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.672 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.672 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:12.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:12.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:12.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:12.672 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:28:12.673 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.673 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.673 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.673 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.673 DEBUG:boto:Method: GET 2024-03-04T16:28:12.673 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.674 DEBUG:boto:Data: 2024-03-04T16:28:12.674 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.674 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.674 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.674 DEBUG:boto:Params: {} 2024-03-04T16:28:12.674 DEBUG:boto:Token: None 2024-03-04T16:28:12.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/?versions 2024-03-04T16:28:12.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08= 2024-03-04T16:28:12.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08=', 'Content-Length': '0'} 2024-03-04T16:28:12.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645acbbe78830fbf-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.677 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.678 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:28:12.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-03-04T16:28:12.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-03-04T16:28:12.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-03-04T16:28:12.678 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.678 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.678 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.678 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.678 DEBUG:boto:Method: GET 2024-03-04T16:28:12.678 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.679 DEBUG:boto:Data: 2024-03-04T16:28:12.679 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.679 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.679 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.679 DEBUG:boto:Params: {} 2024-03-04T16:28:12.679 DEBUG:boto:Token: None 2024-03-04T16:28:12.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/?versions 2024-03-04T16:28:12.679 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08= 2024-03-04T16:28:12.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08=', 'Content-Length': '0'} 2024-03-04T16:28:12.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000919338ceeaa2b9b3-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.682 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.683 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.683 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.683 DEBUG:boto:path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.683 DEBUG:boto:auth_path=/sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.684 DEBUG:boto:Method: GET 2024-03-04T16:28:12.684 DEBUG:boto:Path: /sbnvju-21/?versions&max-keys=999 2024-03-04T16:28:12.684 DEBUG:boto:Data: 2024-03-04T16:28:12.684 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.684 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.684 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.684 DEBUG:boto:Params: {} 2024-03-04T16:28:12.684 DEBUG:boto:Token: None 2024-03-04T16:28:12.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/?versions 2024-03-04T16:28:12.685 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08= 2024-03-04T16:28:12.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fFnZVSfmPHNEVuCF3r0H6l1HJ08=', 'Content-Length': '0'} 2024-03-04T16:28:12.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017afa6e2bb2cc090-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.688 DEBUG:boto:b'sbnvju-21999falsedummyfYDokzjRDpSekb1laNAM2tNJQV0Wv-Ftrue2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1IMNglNnE.XhyagwfPEbS7Rh2R16lFoxtrue2024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2OWB999Dkw0uszwOikn3FFkHqYkuU6wttrue2024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnbtrue2024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:12.688 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.689 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.689 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.689 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.689 DEBUG:boto:Method: GET 2024-03-04T16:28:12.689 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.689 DEBUG:boto:Data: 2024-03-04T16:28:12.689 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.689 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.689 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.689 DEBUG:boto:Params: {} 2024-03-04T16:28:12.690 DEBUG:boto:Token: None 2024-03-04T16:28:12.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f35682619bc92c3-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.693 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.693 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.693 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.693 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.693 DEBUG:boto:Method: GET 2024-03-04T16:28:12.693 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.694 DEBUG:boto:Data: 2024-03-04T16:28:12.694 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.694 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.695 DEBUG:boto:Params: {} 2024-03-04T16:28:12.695 DEBUG:boto:Token: None 2024-03-04T16:28:12.695 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.695 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006698b3424be0488e-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.698 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.698 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.698 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.698 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.698 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.698 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.698 DEBUG:boto:Data: 2024-03-04T16:28:12.698 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.698 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.698 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.698 DEBUG:boto:Params: {} 2024-03-04T16:28:12.699 DEBUG:boto:Token: None 2024-03-04T16:28:12.699 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.699 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qJ4V/J+YJ2s7h7DayQYvsUOFTZ4= 2024-03-04T16:28:12.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qJ4V/J+YJ2s7h7DayQYvsUOFTZ4=', 'Content-Length': '0'} 2024-03-04T16:28:12.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000264741c887434804-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.701 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.701 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.701 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.701 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.702 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.702 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.702 DEBUG:boto:Data: 2024-03-04T16:28:12.702 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.702 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.702 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.702 DEBUG:boto:Params: {} 2024-03-04T16:28:12.703 DEBUG:boto:Token: None 2024-03-04T16:28:12.703 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.703 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qJ4V/J+YJ2s7h7DayQYvsUOFTZ4= 2024-03-04T16:28:12.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qJ4V/J+YJ2s7h7DayQYvsUOFTZ4=', 'Content-Length': '0'} 2024-03-04T16:28:12.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c9c71b8936b4bf2-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:12.705 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.705 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.705 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.705 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.706 DEBUG:boto:Method: GET 2024-03-04T16:28:12.706 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.706 DEBUG:boto:Data: 2024-03-04T16:28:12.706 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.707 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.707 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.707 DEBUG:boto:Params: {} 2024-03-04T16:28:12.707 DEBUG:boto:Token: None 2024-03-04T16:28:12.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.707 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.709 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004cec8fa349e4cd8-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.709 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.709 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.710 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.710 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.710 DEBUG:boto:Method: GET 2024-03-04T16:28:12.710 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.710 DEBUG:boto:Data: 2024-03-04T16:28:12.710 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.710 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.710 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.710 DEBUG:boto:Params: {} 2024-03-04T16:28:12.710 DEBUG:boto:Token: None 2024-03-04T16:28:12.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.711 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012b8d7317f8cdda9-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.713 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.713 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.713 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.713 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.713 DEBUG:boto:Method: GET 2024-03-04T16:28:12.713 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.713 DEBUG:boto:Data: 2024-03-04T16:28:12.714 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.714 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.714 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.714 DEBUG:boto:Params: {} 2024-03-04T16:28:12.714 DEBUG:boto:Token: None 2024-03-04T16:28:12.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0de6a44d0ae777f-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.760 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.761 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.761 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.761 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.761 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.761 DEBUG:boto:Method: GET 2024-03-04T16:28:12.761 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.761 DEBUG:boto:Data: 2024-03-04T16:28:12.761 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.761 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.761 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.761 DEBUG:boto:Params: {} 2024-03-04T16:28:12.762 DEBUG:boto:Token: None 2024-03-04T16:28:12.762 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.762 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea8bcbfd55d82a8-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.768 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.768 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.768 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.769 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.769 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:12.769 DEBUG:boto:Data: 2024-03-04T16:28:12.769 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.769 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.769 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.769 DEBUG:boto:Params: {} 2024-03-04T16:28:12.769 DEBUG:boto:Token: None 2024-03-04T16:28:12.769 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy 2024-03-04T16:28:12.769 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JAYxuDuyrPEA2YPBP21cKzjzA54= 2024-03-04T16:28:12.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JAYxuDuyrPEA2YPBP21cKzjzA54=', 'Content-Length': '0'} 2024-03-04T16:28:12.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a2ea27eaea3bb48-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.772 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.772 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.772 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.772 DEBUG:boto:Path: /sbnvju-21/dummy 2024-03-04T16:28:12.772 DEBUG:boto:Data: 2024-03-04T16:28:12.772 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.772 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.773 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.773 DEBUG:boto:Params: {} 2024-03-04T16:28:12.773 DEBUG:boto:Token: None 2024-03-04T16:28:12.773 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy 2024-03-04T16:28:12.773 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JAYxuDuyrPEA2YPBP21cKzjzA54= 2024-03-04T16:28:12.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JAYxuDuyrPEA2YPBP21cKzjzA54=', 'Content-Length': '0'} 2024-03-04T16:28:12.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004041871d9b166f88-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:28:12.783 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.783 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.784 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.784 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.784 DEBUG:boto:Method: GET 2024-03-04T16:28:12.784 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.784 DEBUG:boto:Data: 2024-03-04T16:28:12.784 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.784 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.784 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.784 DEBUG:boto:Params: {} 2024-03-04T16:28:12.784 DEBUG:boto:Token: None 2024-03-04T16:28:12.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa9caa79fc887175-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.790 DEBUG:boto:path=/sbnvju-21/dummy 2024-03-04T16:28:12.790 DEBUG:boto:auth_path=/sbnvju-21/dummy 2024-03-04T16:28:12.790 DEBUG:boto:path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.790 DEBUG:boto:auth_path=/sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.790 DEBUG:boto:Method: GET 2024-03-04T16:28:12.790 DEBUG:boto:Path: /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.790 DEBUG:boto:Data: 2024-03-04T16:28:12.790 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.791 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.791 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.791 DEBUG:boto:Params: {} 2024-03-04T16:28:12.791 DEBUG:boto:Token: None 2024-03-04T16:28:12.791 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/dummy?versionId=fYDokzjRDpSekb1laNAM2tNJQV0Wv-F 2024-03-04T16:28:12.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw= 2024-03-04T16:28:12.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ocmG++8WcFtzoQivia1Mtsygsxw=', 'Content-Length': '0'} 2024-03-04T16:28:12.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:05 GMT'), ('x-amz-version-id', 'fYDokzjRDpSekb1laNAM2tNJQV0Wv-F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3f5afe0de0811e2-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.795 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.795 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.795 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.795 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.795 DEBUG:boto:Method: GET 2024-03-04T16:28:12.796 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.796 DEBUG:boto:Data: 2024-03-04T16:28:12.796 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.796 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.796 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.796 DEBUG:boto:Params: {} 2024-03-04T16:28:12.799 DEBUG:boto:Token: None 2024-03-04T16:28:12.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.800 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081abaaaf53f93422-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.807 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.808 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.808 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.808 DEBUG:boto:path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.808 DEBUG:boto:auth_path=/sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.808 DEBUG:boto:Method: GET 2024-03-04T16:28:12.808 DEBUG:boto:Path: /sbnvju-21/?prefix=dummy 2024-03-04T16:28:12.808 DEBUG:boto:Data: 2024-03-04T16:28:12.808 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.809 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.809 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.809 DEBUG:boto:Params: {} 2024-03-04T16:28:12.809 DEBUG:boto:Token: None 2024-03-04T16:28:12.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e34a70f4cb6f244-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.812 DEBUG:boto:b'sbnvju-21dummy1000falsedummy2024-03-04T16:25:05.499Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:12.813 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.813 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.813 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.813 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.814 DEBUG:boto:Method: GET 2024-03-04T16:28:12.814 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.814 DEBUG:boto:Data: 2024-03-04T16:28:12.814 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.814 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.814 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.814 DEBUG:boto:Params: {} 2024-03-04T16:28:12.814 DEBUG:boto:Token: None 2024-03-04T16:28:12.815 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.815 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d77efadc4516c6c5-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.817 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.817 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.817 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.817 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.817 DEBUG:boto:Method: GET 2024-03-04T16:28:12.818 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.818 DEBUG:boto:Data: 2024-03-04T16:28:12.818 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.818 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.818 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.818 DEBUG:boto:Params: {} 2024-03-04T16:28:12.818 DEBUG:boto:Token: None 2024-03-04T16:28:12.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.819 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000249b8e7daee2119f-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.865 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.865 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.865 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.865 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.865 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.865 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.865 DEBUG:boto:Data: 2024-03-04T16:28:12.865 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.865 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.866 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.866 DEBUG:boto:Params: {} 2024-03-04T16:28:12.866 DEBUG:boto:Token: None 2024-03-04T16:28:12.866 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.866 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tTkhioSMc0pcC5ysqcP8KE2J+Xs= 2024-03-04T16:28:12.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tTkhioSMc0pcC5ysqcP8KE2J+Xs=', 'Content-Length': '0'} 2024-03-04T16:28:12.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000326c596c851515e1-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.872 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.872 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.872 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.872 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.873 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.873 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.873 DEBUG:boto:Data: 2024-03-04T16:28:12.873 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.873 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.873 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.874 DEBUG:boto:Params: {} 2024-03-04T16:28:12.874 DEBUG:boto:Token: None 2024-03-04T16:28:12.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tTkhioSMc0pcC5ysqcP8KE2J+Xs= 2024-03-04T16:28:12.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tTkhioSMc0pcC5ysqcP8KE2J+Xs=', 'Content-Length': '0'} 2024-03-04T16:28:12.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a9077a19afad7ba-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:12.876 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.877 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.877 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.877 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.877 DEBUG:boto:Method: GET 2024-03-04T16:28:12.877 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.877 DEBUG:boto:Data: 2024-03-04T16:28:12.877 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.878 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.878 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.878 DEBUG:boto:Params: {} 2024-03-04T16:28:12.879 DEBUG:boto:Token: None 2024-03-04T16:28:12.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000098fb5e54d9f0ea6-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.925 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.925 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.925 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.926 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.926 DEBUG:boto:Method: GET 2024-03-04T16:28:12.926 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.926 DEBUG:boto:Data: 2024-03-04T16:28:12.926 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.926 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.926 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.927 DEBUG:boto:Params: {} 2024-03-04T16:28:12.927 DEBUG:boto:Token: None 2024-03-04T16:28:12.927 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.927 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e9f20e467d62943-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.972 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.972 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.972 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.973 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.973 DEBUG:boto:Method: GET 2024-03-04T16:28:12.973 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.973 DEBUG:boto:Data: 2024-03-04T16:28:12.973 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.973 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.973 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.973 DEBUG:boto:Params: {} 2024-03-04T16:28:12.973 DEBUG:boto:Token: None 2024-03-04T16:28:12.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554a7cc745638640-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.977 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.977 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:12.977 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:12.977 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.978 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.978 DEBUG:boto:Method: GET 2024-03-04T16:28:12.978 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:12.978 DEBUG:boto:Data: 2024-03-04T16:28:12.978 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.978 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.978 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.978 DEBUG:boto:Params: {} 2024-03-04T16:28:12.979 DEBUG:boto:Token: None 2024-03-04T16:28:12.979 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/ 2024-03-04T16:28:12.979 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE= 2024-03-04T16:28:12.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JH9Xw/sPm1jGfGtTPkZ2OJhgpBE=', 'Content-Length': '0'} 2024-03-04T16:28:12.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde63a2f3455f035-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.981 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:12.982 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.982 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.982 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.982 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.982 DEBUG:boto:Data: 2024-03-04T16:28:12.982 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.983 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.983 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.983 DEBUG:boto:Params: {} 2024-03-04T16:28:12.983 DEBUG:boto:Token: None 2024-03-04T16:28:12.983 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.983 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t63gyD524xSJO+mqdw+BJ5GMhk0= 2024-03-04T16:28:12.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t63gyD524xSJO+mqdw+BJ5GMhk0=', 'Content-Length': '0'} 2024-03-04T16:28:12.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453921d83ed69e09-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.986 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.986 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.986 DEBUG:boto:Method: HEAD 2024-03-04T16:28:12.986 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.986 DEBUG:boto:Data: 2024-03-04T16:28:12.986 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.986 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:12.986 DEBUG:boto:Port: 8000 2024-03-04T16:28:12.986 DEBUG:boto:Params: {} 2024-03-04T16:28:12.987 DEBUG:boto:Token: None 2024-03-04T16:28:12.987 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t63gyD524xSJO+mqdw+BJ5GMhk0= 2024-03-04T16:28:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t63gyD524xSJO+mqdw+BJ5GMhk0=', 'Content-Length': '0'} 2024-03-04T16:28:12.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b39dec05bb7324f-0065e5f69c-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:12.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-03-04T16:28:12.989 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.989 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:12.990 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.990 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.990 DEBUG:boto:Method: GET 2024-03-04T16:28:12.990 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.990 DEBUG:boto:Data: 2024-03-04T16:28:12.990 DEBUG:boto:Headers: {} 2024-03-04T16:28:12.990 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:12.990 DEBUG:boto:Port: 8001 2024-03-04T16:28:12.990 DEBUG:boto:Params: {} 2024-03-04T16:28:12.990 DEBUG:boto:Token: None 2024-03-04T16:28:12.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:12 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:12.991 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ= 2024-03-04T16:28:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:12 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iwdlW+JDhRR1w0aDXJBJK/VQsIQ=', 'Content-Length': '0'} 2024-03-04T16:28:12.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000920ce68851b4de4f-0065e5f69c-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:12 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.037 DEBUG:boto:path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:13.037 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1 2024-03-04T16:28:13.037 DEBUG:boto:path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:13.037 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:13.037 DEBUG:boto:Method: GET 2024-03-04T16:28:13.037 DEBUG:boto:Path: /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:13.037 DEBUG:boto:Data: 2024-03-04T16:28:13.037 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.038 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.038 DEBUG:boto:Params: {} 2024-03-04T16:28:13.038 DEBUG:boto:Token: None 2024-03-04T16:28:13.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone1?versionId=IMNglNnE.XhyagwfPEbS7Rh2R16lFox 2024-03-04T16:28:13.039 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u91MxaxadYHZbE8nIcjlc4MICBo= 2024-03-04T16:28:13.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u91MxaxadYHZbE8nIcjlc4MICBo=', 'Content-Length': '0'} 2024-03-04T16:28:13.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'IMNglNnE.XhyagwfPEbS7Rh2R16lFox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077188fc0feab1331-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.084 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.084 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.084 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.084 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.085 DEBUG:boto:Method: GET 2024-03-04T16:28:13.085 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.085 DEBUG:boto:Data: 2024-03-04T16:28:13.085 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.085 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.085 DEBUG:boto:Params: {} 2024-03-04T16:28:13.085 DEBUG:boto:Token: None 2024-03-04T16:28:13.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010945bf135ef3951-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.088 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.089 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.089 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.089 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.089 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.089 DEBUG:boto:Method: GET 2024-03-04T16:28:13.089 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone1 2024-03-04T16:28:13.089 DEBUG:boto:Data: 2024-03-04T16:28:13.089 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.089 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.089 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.089 DEBUG:boto:Params: {} 2024-03-04T16:28:13.090 DEBUG:boto:Token: None 2024-03-04T16:28:13.090 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db6d5bf3e3ade218-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.092 DEBUG:boto:b'sbnvju-21obj-test-zone11000falseobj-test-zone12024-03-04T16:25:36.387Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:13.093 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.093 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.093 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.093 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.093 DEBUG:boto:Method: GET 2024-03-04T16:28:13.093 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.093 DEBUG:boto:Data: 2024-03-04T16:28:13.094 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.094 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.094 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.094 DEBUG:boto:Params: {} 2024-03-04T16:28:13.094 DEBUG:boto:Token: None 2024-03-04T16:28:13.094 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.094 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4862bd2b7391ee3-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.140 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.140 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.140 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.140 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.141 DEBUG:boto:Method: GET 2024-03-04T16:28:13.141 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.144 DEBUG:boto:Data: 2024-03-04T16:28:13.144 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.144 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.144 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.144 DEBUG:boto:Params: {} 2024-03-04T16:28:13.145 DEBUG:boto:Token: None 2024-03-04T16:28:13.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccbb91342988f785-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.147 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.147 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.148 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.148 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.148 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.148 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.149 DEBUG:boto:Data: 2024-03-04T16:28:13.149 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.149 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.149 DEBUG:boto:Params: {} 2024-03-04T16:28:13.150 DEBUG:boto:Token: None 2024-03-04T16:28:13.150 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TtEKgCOwdTdfcgm7SjAHl7OWaAw= 2024-03-04T16:28:13.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TtEKgCOwdTdfcgm7SjAHl7OWaAw=', 'Content-Length': '0'} 2024-03-04T16:28:13.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b6d15d4bfb4ac5a-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.152 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.152 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.152 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.152 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.152 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.152 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.152 DEBUG:boto:Data: 2024-03-04T16:28:13.153 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.153 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.153 DEBUG:boto:Params: {} 2024-03-04T16:28:13.153 DEBUG:boto:Token: None 2024-03-04T16:28:13.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.153 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TtEKgCOwdTdfcgm7SjAHl7OWaAw= 2024-03-04T16:28:13.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TtEKgCOwdTdfcgm7SjAHl7OWaAw=', 'Content-Length': '0'} 2024-03-04T16:28:13.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000990995042408c95-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:13.155 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.155 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.155 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.155 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.155 DEBUG:boto:Method: GET 2024-03-04T16:28:13.156 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.156 DEBUG:boto:Data: 2024-03-04T16:28:13.156 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.156 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.156 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.156 DEBUG:boto:Params: {} 2024-03-04T16:28:13.156 DEBUG:boto:Token: None 2024-03-04T16:28:13.156 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.156 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb685fd11536b47b-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.200 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.200 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.200 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.200 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.201 DEBUG:boto:Method: GET 2024-03-04T16:28:13.201 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.201 DEBUG:boto:Data: 2024-03-04T16:28:13.201 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.201 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.201 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.201 DEBUG:boto:Params: {} 2024-03-04T16:28:13.201 DEBUG:boto:Token: None 2024-03-04T16:28:13.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9645d6fb2d6c1da-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.204 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.204 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.204 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.204 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.204 DEBUG:boto:Method: GET 2024-03-04T16:28:13.204 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.205 DEBUG:boto:Data: 2024-03-04T16:28:13.205 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.205 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.205 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.205 DEBUG:boto:Params: {} 2024-03-04T16:28:13.205 DEBUG:boto:Token: None 2024-03-04T16:28:13.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afc1d6a0fedf2f02-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.208 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.209 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.209 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.209 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.209 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.209 DEBUG:boto:Method: GET 2024-03-04T16:28:13.209 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.209 DEBUG:boto:Data: 2024-03-04T16:28:13.209 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.209 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.209 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.210 DEBUG:boto:Params: {} 2024-03-04T16:28:13.210 DEBUG:boto:Token: None 2024-03-04T16:28:13.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63ddf4e772b1406-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.212 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.213 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.213 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.213 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.213 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.213 DEBUG:boto:Data: 2024-03-04T16:28:13.213 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.213 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.214 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.214 DEBUG:boto:Params: {} 2024-03-04T16:28:13.214 DEBUG:boto:Token: None 2024-03-04T16:28:13.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.214 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HxHo0R+7hCiCCdrdGPfLap7Zyyc= 2024-03-04T16:28:13.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HxHo0R+7hCiCCdrdGPfLap7Zyyc=', 'Content-Length': '0'} 2024-03-04T16:28:13.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d89af02fe5b56c8b-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.218 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.219 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.219 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.219 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.219 DEBUG:boto:Data: 2024-03-04T16:28:13.219 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.219 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.219 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.219 DEBUG:boto:Params: {} 2024-03-04T16:28:13.219 DEBUG:boto:Token: None 2024-03-04T16:28:13.219 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.220 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HxHo0R+7hCiCCdrdGPfLap7Zyyc= 2024-03-04T16:28:13.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HxHo0R+7hCiCCdrdGPfLap7Zyyc=', 'Content-Length': '0'} 2024-03-04T16:28:13.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e012593028fd863-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-03-04T16:28:13.222 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.222 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.222 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.222 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.222 DEBUG:boto:Method: GET 2024-03-04T16:28:13.222 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.223 DEBUG:boto:Data: 2024-03-04T16:28:13.223 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.223 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.223 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.223 DEBUG:boto:Params: {} 2024-03-04T16:28:13.223 DEBUG:boto:Token: None 2024-03-04T16:28:13.223 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.223 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000656478d738e2a364-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.269 DEBUG:boto:path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.269 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2 2024-03-04T16:28:13.269 DEBUG:boto:path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.269 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.269 DEBUG:boto:Method: GET 2024-03-04T16:28:13.269 DEBUG:boto:Path: /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.269 DEBUG:boto:Data: 2024-03-04T16:28:13.269 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.269 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.270 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.270 DEBUG:boto:Params: {} 2024-03-04T16:28:13.270 DEBUG:boto:Token: None 2024-03-04T16:28:13.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone2?versionId=OWB999Dkw0uszwOikn3FFkHqYkuU6wt 2024-03-04T16:28:13.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o= 2024-03-04T16:28:13.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JfYAewR7tbitRBFm3u/vRlhR7+o=', 'Content-Length': '0'} 2024-03-04T16:28:13.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'OWB999Dkw0uszwOikn3FFkHqYkuU6wt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd13627853019396-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.272 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.272 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.272 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.272 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.273 DEBUG:boto:Method: GET 2024-03-04T16:28:13.273 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.273 DEBUG:boto:Data: 2024-03-04T16:28:13.273 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.273 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.273 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.273 DEBUG:boto:Params: {} 2024-03-04T16:28:13.273 DEBUG:boto:Token: None 2024-03-04T16:28:13.273 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0aa06515264ee95-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.276 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.276 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.276 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.276 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.276 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.277 DEBUG:boto:Method: GET 2024-03-04T16:28:13.277 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone2 2024-03-04T16:28:13.277 DEBUG:boto:Data: 2024-03-04T16:28:13.277 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.277 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.277 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.277 DEBUG:boto:Params: {} 2024-03-04T16:28:13.277 DEBUG:boto:Token: None 2024-03-04T16:28:13.277 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcd406b114f1386-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.280 DEBUG:boto:b'sbnvju-21obj-test-zone21000falseobj-test-zone22024-03-04T16:25:36.479Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:13.280 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.280 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.281 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.281 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.281 DEBUG:boto:Method: GET 2024-03-04T16:28:13.281 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.281 DEBUG:boto:Data: 2024-03-04T16:28:13.281 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.281 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.281 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.281 DEBUG:boto:Params: {} 2024-03-04T16:28:13.281 DEBUG:boto:Token: None 2024-03-04T16:28:13.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.282 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f09395fa106c112b-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.284 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.284 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.284 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.284 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.284 DEBUG:boto:Method: GET 2024-03-04T16:28:13.284 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.284 DEBUG:boto:Data: 2024-03-04T16:28:13.284 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.284 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.285 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.285 DEBUG:boto:Params: {} 2024-03-04T16:28:13.285 DEBUG:boto:Token: None 2024-03-04T16:28:13.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.285 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a3494bddb52923a-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.328 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.328 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.329 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.329 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.329 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.329 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.329 DEBUG:boto:Data: 2024-03-04T16:28:13.329 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.329 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.329 DEBUG:boto:Params: {} 2024-03-04T16:28:13.329 DEBUG:boto:Token: None 2024-03-04T16:28:13.330 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RaJZvulXk7W/OV+6CW/Xkg9/O4M= 2024-03-04T16:28:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RaJZvulXk7W/OV+6CW/Xkg9/O4M=', 'Content-Length': '0'} 2024-03-04T16:28:13.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0e2313420933bc7-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.332 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.332 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.332 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.332 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.332 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.332 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.332 DEBUG:boto:Data: 2024-03-04T16:28:13.332 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.332 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.332 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.333 DEBUG:boto:Params: {} 2024-03-04T16:28:13.333 DEBUG:boto:Token: None 2024-03-04T16:28:13.333 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RaJZvulXk7W/OV+6CW/Xkg9/O4M= 2024-03-04T16:28:13.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RaJZvulXk7W/OV+6CW/Xkg9/O4M=', 'Content-Length': '0'} 2024-03-04T16:28:13.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f256e1038245a039-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:13.336 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.336 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.337 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.337 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.337 DEBUG:boto:Method: GET 2024-03-04T16:28:13.337 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.337 DEBUG:boto:Data: 2024-03-04T16:28:13.337 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.337 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.337 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.337 DEBUG:boto:Params: {} 2024-03-04T16:28:13.337 DEBUG:boto:Token: None 2024-03-04T16:28:13.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000573f1f4b0543ccc4-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.340 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.340 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.340 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.340 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.340 DEBUG:boto:Method: GET 2024-03-04T16:28:13.340 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.341 DEBUG:boto:Data: 2024-03-04T16:28:13.341 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.341 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.341 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.341 DEBUG:boto:Params: {} 2024-03-04T16:28:13.341 DEBUG:boto:Token: None 2024-03-04T16:28:13.342 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d385cd161ca2f3d0-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.388 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.388 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.388 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.388 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.389 DEBUG:boto:Method: GET 2024-03-04T16:28:13.389 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.389 DEBUG:boto:Data: 2024-03-04T16:28:13.389 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.389 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.389 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.390 DEBUG:boto:Params: {} 2024-03-04T16:28:13.390 DEBUG:boto:Token: None 2024-03-04T16:28:13.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ac2ae7bbb97a05-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.393 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.393 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.394 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.394 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.394 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.394 DEBUG:boto:Method: GET 2024-03-04T16:28:13.394 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.394 DEBUG:boto:Data: 2024-03-04T16:28:13.394 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.394 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.394 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.394 DEBUG:boto:Params: {} 2024-03-04T16:28:13.394 DEBUG:boto:Token: None 2024-03-04T16:28:13.395 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc2fa06004f1090-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.397 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.397 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.397 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.398 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.398 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.398 DEBUG:boto:Data: 2024-03-04T16:28:13.398 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.398 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.398 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.398 DEBUG:boto:Params: {} 2024-03-04T16:28:13.398 DEBUG:boto:Token: None 2024-03-04T16:28:13.398 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a6C6zVYbAIDGztO0dlosOI7OZ4c= 2024-03-04T16:28:13.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a6C6zVYbAIDGztO0dlosOI7OZ4c=', 'Content-Length': '0'} 2024-03-04T16:28:13.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4e713e509fdbc89-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.401 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.401 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.401 DEBUG:boto:Method: HEAD 2024-03-04T16:28:13.401 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.401 DEBUG:boto:Data: 2024-03-04T16:28:13.401 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.401 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.401 DEBUG:boto:Params: {} 2024-03-04T16:28:13.402 DEBUG:boto:Token: None 2024-03-04T16:28:13.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:a6C6zVYbAIDGztO0dlosOI7OZ4c= 2024-03-04T16:28:13.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a6C6zVYbAIDGztO0dlosOI7OZ4c=', 'Content-Length': '0'} 2024-03-04T16:28:13.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d95c58e21630c6c5-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-03-04T16:28:13.404 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.404 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.404 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.404 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.404 DEBUG:boto:Method: GET 2024-03-04T16:28:13.404 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.405 DEBUG:boto:Data: 2024-03-04T16:28:13.405 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.405 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.405 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.405 DEBUG:boto:Params: {} 2024-03-04T16:28:13.405 DEBUG:boto:Token: None 2024-03-04T16:28:13.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cb9a5496d401b83-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.452 DEBUG:boto:path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.452 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3 2024-03-04T16:28:13.452 DEBUG:boto:path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.452 DEBUG:boto:auth_path=/sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.453 DEBUG:boto:Method: GET 2024-03-04T16:28:13.453 DEBUG:boto:Path: /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.453 DEBUG:boto:Data: 2024-03-04T16:28:13.453 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.453 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.453 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.453 DEBUG:boto:Params: {} 2024-03-04T16:28:13.453 DEBUG:boto:Token: None 2024-03-04T16:28:13.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/obj-test-zone3?versionId=kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb 2024-03-04T16:28:13.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g= 2024-03-04T16:28:13.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GGxeaJ5ACf75T+f7e+M1ROvDx0g=', 'Content-Length': '0'} 2024-03-04T16:28:13.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:25:36 GMT'), ('x-amz-version-id', 'kKNJa3ohi7T-ia5WWbnSyKpDQwYGfnb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-21:fc294220-5038-4f25-aeee-cac2c5838729.4341.30'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000244bb6733c403ec4-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.500 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.500 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.501 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.501 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.501 DEBUG:boto:Method: GET 2024-03-04T16:28:13.501 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.501 DEBUG:boto:Data: 2024-03-04T16:28:13.501 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:28:13.501 DEBUG:boto:Port: 8001 2024-03-04T16:28:13.501 DEBUG:boto:Params: {} 2024-03-04T16:28:13.501 DEBUG:boto:Token: None 2024-03-04T16:28:13.502 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c77ea0b176ff5dd-0065e5f69d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.505 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.505 DEBUG:boto:path=/sbnvju-21/ 2024-03-04T16:28:13.505 DEBUG:boto:auth_path=/sbnvju-21/ 2024-03-04T16:28:13.505 DEBUG:boto:path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.505 DEBUG:boto:auth_path=/sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.506 DEBUG:boto:Method: GET 2024-03-04T16:28:13.506 DEBUG:boto:Path: /sbnvju-21/?prefix=obj-test-zone3 2024-03-04T16:28:13.506 DEBUG:boto:Data: 2024-03-04T16:28:13.506 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.506 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:28:13.506 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.506 DEBUG:boto:Params: {} 2024-03-04T16:28:13.506 DEBUG:boto:Token: None 2024-03-04T16:28:13.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-21/ 2024-03-04T16:28:13.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg= 2024-03-04T16:28:13.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RgrzVbho1muSe/O4SXsLsRqwJMg=', 'Content-Length': '0'} 2024-03-04T16:28:13.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d950f82d41f417-0065e5f69d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.552 DEBUG:boto:b'sbnvju-21obj-test-zone31000falseobj-test-zone32024-03-04T16:25:36.576Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:28:13.553 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-21 zones={test-zone3, test-zone2} 2024-03-04T16:28:13.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-03-04T16:28:13.556 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:28:13.557 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:28:13.557 DEBUG:boto:Method: PUT 2024-03-04T16:28:13.557 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:28:13.557 DEBUG:boto:Data: 2024-03-04T16:28:13.557 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:13.557 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.557 DEBUG:boto:Params: {} 2024-03-04T16:28:13.557 DEBUG:boto:Token: None 2024-03-04T16:28:13.558 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-22/ 2024-03-04T16:28:13.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VsLq7mDUXAjB7+00llaZct5D9oY= 2024-03-04T16:28:13.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VsLq7mDUXAjB7+00llaZct5D9oY=', 'Content-Length': '0'} 2024-03-04T16:28:13.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000797b101596cf6b90-0065e5f69d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.575 DEBUG:rgw_multi.tests:created bucket=sbnvju-22 2024-03-04T16:28:13.575 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:28:13.575 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:28:13.575 DEBUG:boto:path=/sbnvju-22/?versioning 2024-03-04T16:28:13.575 DEBUG:boto:auth_path=/sbnvju-22/?versioning 2024-03-04T16:28:13.575 DEBUG:boto:Method: PUT 2024-03-04T16:28:13.575 DEBUG:boto:Path: /sbnvju-22/?versioning 2024-03-04T16:28:13.575 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:28:13.575 DEBUG:boto:Headers: {} 2024-03-04T16:28:13.576 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:13.576 DEBUG:boto:Port: 8000 2024-03-04T16:28:13.576 DEBUG:boto:Params: {} 2024-03-04T16:28:13.576 DEBUG:boto:Token: None 2024-03-04T16:28:13.576 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:28:13 GMT /sbnvju-22/?versioning 2024-03-04T16:28:13.576 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qeuHjUdQbuf3oxgTNosg0e02/dg= 2024-03-04T16:28:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:13 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qeuHjUdQbuf3oxgTNosg0e02/dg=', 'Content-Length': '228'} 2024-03-04T16:28:13.584 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e97478ba1575d55d-0065e5f69d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:13 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:13.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:13.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:13.585 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:28:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:13.638 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:13.687 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:28:13.707 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709569693.582430_212.1","last_update":"2024-03-04T16:28:13.582430Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:28:13.708 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:13.708 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:28:13.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:13.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:13.709 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:28:13.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:13.757 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:13.805 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:13.805+0000 7f390c816800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:13.807 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:13.806+0000 7f390c816800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:13.826 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.565981_207.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.565981Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:13Z"} 2024-03-04T16:28:13.826 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.565981_207.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.565981Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:13Z"} 2024-03-04T16:28:13.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569693.565981_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:28:13.565981Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:28:13.826 DEBUG:rgw_multi.tests:log_status={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:13.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569693.565981_207.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:13.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709569693.582430_212.1 target=1_1709569693.565981_207.1 2024-03-04T16:28:18.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:18.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:18.828 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:28:18.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:18.884 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:18.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:18.935+0000 7f27eed2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:18.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:18.936+0000 7f27eed2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:18.955 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.582430_212.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.582430Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:18Z"} 2024-03-04T16:28:18.956 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.582430_212.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.582430Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:18Z"} 2024-03-04T16:28:18.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569693.582430_212.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:28:13.582430Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:28:18.956 DEBUG:rgw_multi.tests:log_status={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:18.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:18.956 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:28:18.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:28:18.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:18.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:28:18.957 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:28:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:19.061+0000 7fc44073c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:19.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:19.062+0000 7fc44073c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:28:19.083 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.582430_212.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.582430Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:19Z"} 2024-03-04T16:28:19.083 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569693.582430_212.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:28:13.582430Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:28:19Z"} 2024-03-04T16:28:19.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569693.582430_212.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:28:13.582430Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:28:19.084 DEBUG:rgw_multi.tests:log_status={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:19.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1709569693.582430_212.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:28:19.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:28:19.084 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:28:19.084 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:28:19.084 DEBUG:boto:Method: HEAD 2024-03-04T16:28:19.084 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:28:19.085 DEBUG:boto:Data: 2024-03-04T16:28:19.085 DEBUG:boto:Headers: {} 2024-03-04T16:28:19.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:19.085 DEBUG:boto:Port: 8000 2024-03-04T16:28:19.085 DEBUG:boto:Params: {} 2024-03-04T16:28:19.085 DEBUG:boto:Token: None 2024-03-04T16:28:19.085 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:28:19 GMT /sbnvju-22/ 2024-03-04T16:28:19.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8/L5tSEsK23t5euQgmGQ37HPDj4= 2024-03-04T16:28:19.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:28:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8/L5tSEsK23t5euQgmGQ37HPDj4=', 'Content-Length': '0'} 2024-03-04T16:28:19.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007b35527fe6a5f1c-0065e5f6a3-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:19.088 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:28:19.089 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:28:19.089 DEBUG:boto:Method: PUT 2024-03-04T16:28:19.089 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:28:19.089 DEBUG:boto:Data: 2024-03-04T16:28:19.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:28:19.089 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:28:19.089 DEBUG:boto:Port: 8000 2024-03-04T16:28:19.089 DEBUG:boto:Params: {} 2024-03-04T16:28:19.089 DEBUG:boto:Token: None 2024-03-04T16:28:19.089 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:28:19 GMT /sbnvju-22/dummy 2024-03-04T16:28:19.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H/eqMTtD4xlwCf3pEzxQC0q7rBk= 2024-03-04T16:28:19.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:28:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H/eqMTtD4xlwCf3pEzxQC0q7rBk='} 2024-03-04T16:28:19.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-amz-request-id', 'tx000003f17eac27a0c27bc-0065e5f6a3-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:28:19.133 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-22 2024-03-04T16:28:19.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:19.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:19.133 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:19.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:19.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:19.234 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:19.234+0000 7fe98b511800 1 waiting to reach incremental sync.. 2024-03-04T16:28:24.235 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:24.235+0000 7fe98b511800 1 waiting to reach incremental sync.. 2024-03-04T16:28:29.236 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:29.236+0000 7fe98b511800 1 waiting to reach incremental sync.. 2024-03-04T16:28:34.236 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:34.236+0000 7fe98b511800 1 waiting to reach incremental sync.. 2024-03-04T16:28:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:39.238+0000 7fe98b511800 1 bucket sync caught up with source: 2024-03-04T16:28:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000003.26.10, , , , , ] 2024-03-04T16:28:39.238 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000002.24.11, , , , , ] 2024-03-04T16:28:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:28:39.238+0000 7fe98b511800 0 bucket checkpoint complete 2024-03-04T16:28:39.261 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-22 2024-03-04T16:28:39.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:39.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:39.261 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:39.374+0000 7f01ae439800 1 waiting for incremental sync to catch up: 2024-03-04T16:28:39.375 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.24.11, , , , , ] 2024-03-04T16:28:39.375 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.26.10, , , , , ] 2024-03-04T16:28:44.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:44.376+0000 7f01ae439800 1 waiting for incremental sync to catch up: 2024-03-04T16:28:44.376 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.24.11, , , , , ] 2024-03-04T16:28:44.377 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.26.10, , , , , ] 2024-03-04T16:28:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:49.377+0000 7f01ae439800 1 bucket sync caught up with source: 2024-03-04T16:28:49.382 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.26.10, , , , , ] 2024-03-04T16:28:49.382 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.26.10, , , , , ] 2024-03-04T16:28:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:49.377+0000 7f01ae439800 0 bucket checkpoint complete 2024-03-04T16:28:49.403 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-22 2024-03-04T16:28:49.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:49.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:49.404 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:49.466 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:49.466 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:49.519+0000 7f9758613800 1 waiting to reach incremental sync.. 2024-03-04T16:28:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:54.520+0000 7f9758613800 1 waiting to reach incremental sync.. 2024-03-04T16:28:59.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:59.522+0000 7f9758613800 1 bucket sync caught up with source: 2024-03-04T16:28:59.523 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , , , ] 2024-03-04T16:28:59.523 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , , , ] 2024-03-04T16:28:59.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:59.522+0000 7f9758613800 0 bucket checkpoint complete 2024-03-04T16:28:59.544 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-22 2024-03-04T16:28:59.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:59.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:59.545 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:59.602 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:59.603 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:59.655+0000 7f58c952e800 1 bucket sync caught up with source: 2024-03-04T16:28:59.655 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , , , ] 2024-03-04T16:28:59.656 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , , , ] 2024-03-04T16:28:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:59.655+0000 7f58c952e800 0 bucket checkpoint complete 2024-03-04T16:28:59.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-22 2024-03-04T16:28:59.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:59.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:28:59.674 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:28:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:28:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:28:59.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:28:59.778+0000 7f60b7430800 1 waiting for incremental sync to catch up: 2024-03-04T16:28:59.778 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-03-04T16:28:59.779 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.33.10, , , , , ] 2024-03-04T16:29:04.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:04.779+0000 7f60b7430800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:04.780 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-03-04T16:29:04.780 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.33.10, , , , , ] 2024-03-04T16:29:09.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:09.781+0000 7f60b7430800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:09.781 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-03-04T16:29:09.782 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.33.10, , , , , ] 2024-03-04T16:29:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:14.782+0000 7f60b7430800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:14.783 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-03-04T16:29:14.783 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.33.10, , , , , ] 2024-03-04T16:29:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:19.784+0000 7f60b7430800 1 bucket sync caught up with source: 2024-03-04T16:29:19.784 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000004.35.10, , , , , ] 2024-03-04T16:29:19.784 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.33.10, , , , , ] 2024-03-04T16:29:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:19.784+0000 7f60b7430800 0 bucket checkpoint complete 2024-03-04T16:29:19.809 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-22 2024-03-04T16:29:19.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:19.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:19.810 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:29:19.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:29:19.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:29:19.921 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:19.921+0000 7fd60ea2e800 1 bucket sync caught up with source: 2024-03-04T16:29:19.947 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000004.35.10, , , , , ] 2024-03-04T16:29:19.948 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000004.35.10, , , , , ] 2024-03-04T16:29:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:19.921+0000 7fd60ea2e800 0 bucket checkpoint complete 2024-03-04T16:29:19.949 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone2, test-zone1} 2024-03-04T16:29:19.949 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.949 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.949 DEBUG:boto:Method: HEAD 2024-03-04T16:29:19.950 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:19.950 DEBUG:boto:Data: 2024-03-04T16:29:19.950 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.950 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:19.950 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.950 DEBUG:boto:Params: {} 2024-03-04T16:29:19.950 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:29:19.951 DEBUG:boto:Token: None 2024-03-04T16:29:19.951 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/ 2024-03-04T16:29:19.951 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EuLzCi9qgfyrtz6GQVl2XYLRe8A= 2024-03-04T16:29:19.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EuLzCi9qgfyrtz6GQVl2XYLRe8A=', 'Content-Length': '0'} 2024-03-04T16:29:19.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000398e0a84d271d7c7-0065e5f6df-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.956 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.956 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.956 DEBUG:boto:Method: HEAD 2024-03-04T16:29:19.956 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:19.956 DEBUG:boto:Data: 2024-03-04T16:29:19.957 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:19.957 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.957 DEBUG:boto:Params: {} 2024-03-04T16:29:19.957 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:29:19.957 DEBUG:boto:Token: None 2024-03-04T16:29:19.957 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/ 2024-03-04T16:29:19.957 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EuLzCi9qgfyrtz6GQVl2XYLRe8A= 2024-03-04T16:29:19.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EuLzCi9qgfyrtz6GQVl2XYLRe8A=', 'Content-Length': '0'} 2024-03-04T16:29:19.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b676149baab09171-0065e5f6df-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:29:19.962 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.962 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.962 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.962 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.963 DEBUG:boto:Method: GET 2024-03-04T16:29:19.963 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.963 DEBUG:boto:Data: 2024-03-04T16:29:19.963 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:19.963 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.963 DEBUG:boto:Params: {} 2024-03-04T16:29:19.963 DEBUG:boto:Token: None 2024-03-04T16:29:19.963 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/?versions 2024-03-04T16:29:19.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk= 2024-03-04T16:29:19.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk=', 'Content-Length': '0'} 2024-03-04T16:29:19.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b1fe523a97a05e1-0065e5f6df-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.966 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:19.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:19.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:29:19.967 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.967 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.967 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.968 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.968 DEBUG:boto:Method: GET 2024-03-04T16:29:19.968 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.968 DEBUG:boto:Data: 2024-03-04T16:29:19.968 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.968 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:19.968 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.968 DEBUG:boto:Params: {} 2024-03-04T16:29:19.968 DEBUG:boto:Token: None 2024-03-04T16:29:19.968 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/?versions 2024-03-04T16:29:19.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk= 2024-03-04T16:29:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk=', 'Content-Length': '0'} 2024-03-04T16:29:19.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94900886a62cc33-0065e5f6df-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.971 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:19.972 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:19.972 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.972 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.973 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.973 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.973 DEBUG:boto:Method: GET 2024-03-04T16:29:19.973 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.973 DEBUG:boto:Data: 2024-03-04T16:29:19.973 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:19.973 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.973 DEBUG:boto:Params: {} 2024-03-04T16:29:19.973 DEBUG:boto:Token: None 2024-03-04T16:29:19.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/?versions 2024-03-04T16:29:19.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk= 2024-03-04T16:29:19.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk=', 'Content-Length': '0'} 2024-03-04T16:29:19.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028593031cd71ef7d-0065e5f6df-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.976 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:19.976 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:19.976 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:19.976 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.977 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.977 DEBUG:boto:Method: GET 2024-03-04T16:29:19.977 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:19.977 DEBUG:boto:Data: 2024-03-04T16:29:19.977 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:19.977 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.977 DEBUG:boto:Params: {} 2024-03-04T16:29:19.977 DEBUG:boto:Token: None 2024-03-04T16:29:19.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/?versions 2024-03-04T16:29:19.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk= 2024-03-04T16:29:19.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R8TKYUKvgGnKg7OuTMrckWTZtXk=', 'Content-Length': '0'} 2024-03-04T16:29:19.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9e09c8d8b268cf-0065e5f6df-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.980 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:19.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:19.981 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:19.981 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:19.981 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.981 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.981 DEBUG:boto:Method: GET 2024-03-04T16:29:19.981 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.981 DEBUG:boto:Data: 2024-03-04T16:29:19.982 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:19.982 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.982 DEBUG:boto:Params: {} 2024-03-04T16:29:19.982 DEBUG:boto:Token: None 2024-03-04T16:29:19.982 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.983 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLMNQqHlk/rhFAgHfVNaesPXYA= 2024-03-04T16:29:19.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLMNQqHlk/rhFAgHfVNaesPXYA=', 'Content-Length': '0'} 2024-03-04T16:29:19.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006659281ab54cd862-0065e5f6df-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.987 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:19.987 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:19.988 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.988 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.988 DEBUG:boto:Method: GET 2024-03-04T16:29:19.988 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.988 DEBUG:boto:Data: 2024-03-04T16:29:19.988 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:19.989 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.989 DEBUG:boto:Params: {} 2024-03-04T16:29:19.989 DEBUG:boto:Token: None 2024-03-04T16:29:19.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLMNQqHlk/rhFAgHfVNaesPXYA= 2024-03-04T16:29:19.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLMNQqHlk/rhFAgHfVNaesPXYA=', 'Content-Length': '0'} 2024-03-04T16:29:19.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b091c6e5a311d99-0065e5f6df-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.994 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:19.994 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:19.995 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.995 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.995 DEBUG:boto:Method: HEAD 2024-03-04T16:29:19.995 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.995 DEBUG:boto:Data: 2024-03-04T16:29:19.995 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.995 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:19.995 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.995 DEBUG:boto:Params: {} 2024-03-04T16:29:19.995 DEBUG:boto:Token: None 2024-03-04T16:29:19.995 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.996 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A3YwLEdeA9iQ3mWdQ8+u9dYzzPM= 2024-03-04T16:29:19.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A3YwLEdeA9iQ3mWdQ8+u9dYzzPM=', 'Content-Length': '0'} 2024-03-04T16:29:19.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d605fe64ef01281-0065e5f6df-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:19.998 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:19.998 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:19.998 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.998 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.998 DEBUG:boto:Method: HEAD 2024-03-04T16:29:19.998 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.998 DEBUG:boto:Data: 2024-03-04T16:29:19.998 DEBUG:boto:Headers: {} 2024-03-04T16:29:19.998 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:19.998 DEBUG:boto:Port: 8000 2024-03-04T16:29:19.998 DEBUG:boto:Params: {} 2024-03-04T16:29:19.999 DEBUG:boto:Token: None 2024-03-04T16:29:19.999 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:19 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:19.999 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A3YwLEdeA9iQ3mWdQ8+u9dYzzPM= 2024-03-04T16:29:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A3YwLEdeA9iQ3mWdQ8+u9dYzzPM=', 'Content-Length': '0'} 2024-03-04T16:29:20.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a27035bedff2b85f-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.002 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.002 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.002 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.002 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.002 DEBUG:boto:Method: GET 2024-03-04T16:29:20.002 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.002 DEBUG:boto:Data: 2024-03-04T16:29:20.002 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.002 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.002 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.003 DEBUG:boto:Params: {} 2024-03-04T16:29:20.003 DEBUG:boto:Token: None 2024-03-04T16:29:20.003 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.003 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000896fac155f59173b-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.005 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.005 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.005 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.005 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.005 DEBUG:boto:Method: GET 2024-03-04T16:29:20.006 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.006 DEBUG:boto:Data: 2024-03-04T16:29:20.006 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.006 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.006 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.006 DEBUG:boto:Params: {} 2024-03-04T16:29:20.006 DEBUG:boto:Token: None 2024-03-04T16:29:20.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.007 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044b5a4bccfcf610f-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.009 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.009 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.009 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.009 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.009 DEBUG:boto:Method: GET 2024-03-04T16:29:20.009 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.010 DEBUG:boto:Data: 2024-03-04T16:29:20.010 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.010 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.010 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.010 DEBUG:boto:Params: {} 2024-03-04T16:29:20.010 DEBUG:boto:Token: None 2024-03-04T16:29:20.011 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.011 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026026088b4166b3d-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.014 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.014 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.014 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.014 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.014 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.015 DEBUG:boto:Method: GET 2024-03-04T16:29:20.015 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.015 DEBUG:boto:Data: 2024-03-04T16:29:20.015 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.015 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.015 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.015 DEBUG:boto:Params: {} 2024-03-04T16:29:20.015 DEBUG:boto:Token: None 2024-03-04T16:29:20.015 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2145cbfc575a96-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.060 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.060 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.061 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.061 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.061 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.061 DEBUG:boto:Data: 2024-03-04T16:29:20.061 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.061 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.061 DEBUG:boto:Params: {} 2024-03-04T16:29:20.061 DEBUG:boto:Token: None 2024-03-04T16:29:20.062 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e35b931f14c51f4e-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.064 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.064 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.064 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.065 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.065 DEBUG:boto:Data: 2024-03-04T16:29:20.065 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.065 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.065 DEBUG:boto:Params: {} 2024-03-04T16:29:20.066 DEBUG:boto:Token: None 2024-03-04T16:29:20.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f87a2968569ad3c-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.069 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.069 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.069 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.069 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.069 DEBUG:boto:Method: GET 2024-03-04T16:29:20.069 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.070 DEBUG:boto:Data: 2024-03-04T16:29:20.070 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.070 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.070 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.070 DEBUG:boto:Params: {} 2024-03-04T16:29:20.070 DEBUG:boto:Token: None 2024-03-04T16:29:20.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d630f7aaf8e7849f-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.074 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.074 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.075 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.075 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.075 DEBUG:boto:Method: GET 2024-03-04T16:29:20.075 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.075 DEBUG:boto:Data: 2024-03-04T16:29:20.075 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.075 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.075 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.075 DEBUG:boto:Params: {} 2024-03-04T16:29:20.076 DEBUG:boto:Token: None 2024-03-04T16:29:20.076 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.076 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003e34e3d70be5bea-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.079 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.079 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.080 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.080 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.080 DEBUG:boto:Method: GET 2024-03-04T16:29:20.080 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.080 DEBUG:boto:Data: 2024-03-04T16:29:20.080 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.081 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.081 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.081 DEBUG:boto:Params: {} 2024-03-04T16:29:20.081 DEBUG:boto:Token: None 2024-03-04T16:29:20.081 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000072415d38799d29c-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.128 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.129 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.129 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.129 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.129 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.129 DEBUG:boto:Method: GET 2024-03-04T16:29:20.137 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.137 DEBUG:boto:Data: 2024-03-04T16:29:20.138 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.138 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.138 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.138 DEBUG:boto:Params: {} 2024-03-04T16:29:20.138 DEBUG:boto:Token: None 2024-03-04T16:29:20.138 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.139 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704ae9b7cb42729d-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.143 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.143 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone2, test-zone1} 2024-03-04T16:29:20.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone3, test-zone1} 2024-03-04T16:29:20.144 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.144 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.144 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.144 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.144 DEBUG:boto:Data: 2024-03-04T16:29:20.144 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.144 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.144 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.145 DEBUG:boto:Params: {} 2024-03-04T16:29:20.145 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:29:20.145 DEBUG:boto:Token: None 2024-03-04T16:29:20.145 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cb230b4da6f01fe-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.152 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.152 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.152 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.152 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.152 DEBUG:boto:Data: 2024-03-04T16:29:20.152 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.152 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.152 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.152 DEBUG:boto:Params: {} 2024-03-04T16:29:20.153 DEBUG:boto:Token: None 2024-03-04T16:29:20.153 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.153 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1195a3eca85681f-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:29:20.156 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.157 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.157 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.157 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.158 DEBUG:boto:Method: GET 2024-03-04T16:29:20.158 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.158 DEBUG:boto:Data: 2024-03-04T16:29:20.158 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.158 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.158 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.158 DEBUG:boto:Params: {} 2024-03-04T16:29:20.159 DEBUG:boto:Token: None 2024-03-04T16:29:20.159 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037f262c9b49ea913-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.163 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.163 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:20.163 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:29:20.163 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.163 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.163 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.164 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.164 DEBUG:boto:Method: GET 2024-03-04T16:29:20.164 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.164 DEBUG:boto:Data: 2024-03-04T16:29:20.164 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.164 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.164 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.164 DEBUG:boto:Params: {} 2024-03-04T16:29:20.164 DEBUG:boto:Token: None 2024-03-04T16:29:20.164 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba077a5799c73172-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.208 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.209 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:20.209 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.209 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.209 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.209 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.209 DEBUG:boto:Method: GET 2024-03-04T16:29:20.209 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.209 DEBUG:boto:Data: 2024-03-04T16:29:20.210 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.210 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.210 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.210 DEBUG:boto:Params: {} 2024-03-04T16:29:20.210 DEBUG:boto:Token: None 2024-03-04T16:29:20.211 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.211 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b234af47f8ef46cb-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.213 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.214 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.214 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.214 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.214 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.214 DEBUG:boto:Method: GET 2024-03-04T16:29:20.214 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.214 DEBUG:boto:Data: 2024-03-04T16:29:20.214 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.214 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.215 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.215 DEBUG:boto:Params: {} 2024-03-04T16:29:20.215 DEBUG:boto:Token: None 2024-03-04T16:29:20.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.215 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c657d054c500d1dc-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.217 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.218 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.218 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.218 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.218 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.218 DEBUG:boto:Method: GET 2024-03-04T16:29:20.218 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.218 DEBUG:boto:Data: 2024-03-04T16:29:20.218 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.218 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.219 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.219 DEBUG:boto:Params: {} 2024-03-04T16:29:20.219 DEBUG:boto:Token: None 2024-03-04T16:29:20.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.219 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000515af90911bf822-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.221 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.221 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.221 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.221 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.222 DEBUG:boto:Method: GET 2024-03-04T16:29:20.222 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.222 DEBUG:boto:Data: 2024-03-04T16:29:20.222 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.222 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.222 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.222 DEBUG:boto:Params: {} 2024-03-04T16:29:20.222 DEBUG:boto:Token: None 2024-03-04T16:29:20.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6b963ec0fbb5e6c-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.229 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.229 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.229 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.229 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.229 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.229 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.229 DEBUG:boto:Data: 2024-03-04T16:29:20.229 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.229 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.230 DEBUG:boto:Params: {} 2024-03-04T16:29:20.230 DEBUG:boto:Token: None 2024-03-04T16:29:20.230 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE= 2024-03-04T16:29:20.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE=', 'Content-Length': '0'} 2024-03-04T16:29:20.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000403edc833d040f5c-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.234 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.234 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.235 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.235 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.235 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.235 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.235 DEBUG:boto:Data: 2024-03-04T16:29:20.235 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.235 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.235 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.235 DEBUG:boto:Params: {} 2024-03-04T16:29:20.236 DEBUG:boto:Token: None 2024-03-04T16:29:20.236 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.236 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE= 2024-03-04T16:29:20.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE=', 'Content-Length': '0'} 2024-03-04T16:29:20.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084cb07e247bb4114-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.238 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.238 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.239 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.239 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.239 DEBUG:boto:Method: GET 2024-03-04T16:29:20.239 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.239 DEBUG:boto:Data: 2024-03-04T16:29:20.239 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.240 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.240 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.240 DEBUG:boto:Params: {} 2024-03-04T16:29:20.240 DEBUG:boto:Token: None 2024-03-04T16:29:20.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6cea4ebf6242f1a-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.244 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.244 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.244 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.244 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.244 DEBUG:boto:Method: GET 2024-03-04T16:29:20.244 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.244 DEBUG:boto:Data: 2024-03-04T16:29:20.245 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.245 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.245 DEBUG:boto:Params: {} 2024-03-04T16:29:20.245 DEBUG:boto:Token: None 2024-03-04T16:29:20.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce6911ed6e8f1b5d-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.247 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.247 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.247 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.247 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.247 DEBUG:boto:Method: GET 2024-03-04T16:29:20.248 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.248 DEBUG:boto:Data: 2024-03-04T16:29:20.248 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.248 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.248 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.248 DEBUG:boto:Params: {} 2024-03-04T16:29:20.248 DEBUG:boto:Token: None 2024-03-04T16:29:20.248 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.248 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000403daaf3a04f4d60-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.292 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.293 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.293 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.293 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.293 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.293 DEBUG:boto:Method: GET 2024-03-04T16:29:20.293 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.294 DEBUG:boto:Data: 2024-03-04T16:29:20.294 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.294 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.294 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.294 DEBUG:boto:Params: {} 2024-03-04T16:29:20.294 DEBUG:boto:Token: None 2024-03-04T16:29:20.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.295 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ebd6a2490126159-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.297 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.298 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.298 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.298 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.298 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.298 DEBUG:boto:Data: 2024-03-04T16:29:20.298 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.299 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.299 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.299 DEBUG:boto:Params: {} 2024-03-04T16:29:20.299 DEBUG:boto:Token: None 2024-03-04T16:29:20.299 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.299 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002be6e7b6d952503a-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.302 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.303 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.303 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.303 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.303 DEBUG:boto:Data: 2024-03-04T16:29:20.304 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.304 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.304 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.304 DEBUG:boto:Params: {} 2024-03-04T16:29:20.305 DEBUG:boto:Token: None 2024-03-04T16:29:20.306 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000452458bb95eae637-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.309 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.310 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.310 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.310 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.310 DEBUG:boto:Method: GET 2024-03-04T16:29:20.310 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.311 DEBUG:boto:Data: 2024-03-04T16:29:20.311 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.311 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.312 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.312 DEBUG:boto:Params: {} 2024-03-04T16:29:20.312 DEBUG:boto:Token: None 2024-03-04T16:29:20.312 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.313 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c95296d775150b5-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.316 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.316 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.316 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.317 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.317 DEBUG:boto:Method: GET 2024-03-04T16:29:20.317 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.317 DEBUG:boto:Data: 2024-03-04T16:29:20.317 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.317 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.318 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.318 DEBUG:boto:Params: {} 2024-03-04T16:29:20.318 DEBUG:boto:Token: None 2024-03-04T16:29:20.318 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.319 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ab31ebe20831e13-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.321 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.321 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.322 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.322 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.322 DEBUG:boto:Method: GET 2024-03-04T16:29:20.322 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.322 DEBUG:boto:Data: 2024-03-04T16:29:20.322 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.322 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.322 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.322 DEBUG:boto:Params: {} 2024-03-04T16:29:20.322 DEBUG:boto:Token: None 2024-03-04T16:29:20.322 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff3caeecd511af73-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.328 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.328 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.329 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.329 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.329 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.329 DEBUG:boto:Method: GET 2024-03-04T16:29:20.329 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.329 DEBUG:boto:Data: 2024-03-04T16:29:20.329 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.329 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.329 DEBUG:boto:Params: {} 2024-03-04T16:29:20.330 DEBUG:boto:Token: None 2024-03-04T16:29:20.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008774a9b25a9179ef-0065e5f6e0-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.376 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.377 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone3, test-zone1} 2024-03-04T16:29:20.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone3, test-zone2} 2024-03-04T16:29:20.377 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.377 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.377 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.377 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.377 DEBUG:boto:Data: 2024-03-04T16:29:20.377 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.377 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.378 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.378 DEBUG:boto:Params: {} 2024-03-04T16:29:20.378 DEBUG:boto:Token: None 2024-03-04T16:29:20.378 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000307f83544510e8f8-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.382 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.382 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.382 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.382 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.382 DEBUG:boto:Data: 2024-03-04T16:29:20.382 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.382 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.382 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.382 DEBUG:boto:Params: {} 2024-03-04T16:29:20.383 DEBUG:boto:Token: None 2024-03-04T16:29:20.383 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.383 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b738091d22df4281-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:29:20.385 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.385 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.386 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.386 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.386 DEBUG:boto:Method: GET 2024-03-04T16:29:20.386 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.386 DEBUG:boto:Data: 2024-03-04T16:29:20.386 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.386 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.386 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.387 DEBUG:boto:Params: {} 2024-03-04T16:29:20.387 DEBUG:boto:Token: None 2024-03-04T16:29:20.387 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.387 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402e64bb4af6177b-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.390 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:20.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:29:20.391 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.391 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.391 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.391 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.391 DEBUG:boto:Method: GET 2024-03-04T16:29:20.391 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.391 DEBUG:boto:Data: 2024-03-04T16:29:20.391 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.391 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.391 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.391 DEBUG:boto:Params: {} 2024-03-04T16:29:20.392 DEBUG:boto:Token: None 2024-03-04T16:29:20.392 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.392 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950186e05a7842c7-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.395 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.395 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:29:20.395 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.395 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.395 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.395 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.396 DEBUG:boto:Method: GET 2024-03-04T16:29:20.396 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.396 DEBUG:boto:Data: 2024-03-04T16:29:20.396 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.396 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.396 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.396 DEBUG:boto:Params: {} 2024-03-04T16:29:20.396 DEBUG:boto:Token: None 2024-03-04T16:29:20.396 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a40d0377148b9adc-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.440 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.440 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.441 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.441 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.441 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.441 DEBUG:boto:Method: GET 2024-03-04T16:29:20.441 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:29:20.441 DEBUG:boto:Data: 2024-03-04T16:29:20.441 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.441 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.441 DEBUG:boto:Params: {} 2024-03-04T16:29:20.442 DEBUG:boto:Token: None 2024-03-04T16:29:20.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/?versions 2024-03-04T16:29:20.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY= 2024-03-04T16:29:20.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJwsZfe/Cc2URqnfxWKm3tuChDY=', 'Content-Length': '0'} 2024-03-04T16:29:20.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f9c7689541dcc3-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.444 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.445 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.445 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.445 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.445 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.445 DEBUG:boto:Method: GET 2024-03-04T16:29:20.445 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.445 DEBUG:boto:Data: 2024-03-04T16:29:20.445 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.445 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.446 DEBUG:boto:Params: {} 2024-03-04T16:29:20.446 DEBUG:boto:Token: None 2024-03-04T16:29:20.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d248cadd08dd2311-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.453 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.453 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.453 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.453 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.453 DEBUG:boto:Method: GET 2024-03-04T16:29:20.453 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.453 DEBUG:boto:Data: 2024-03-04T16:29:20.453 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.454 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.454 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.454 DEBUG:boto:Params: {} 2024-03-04T16:29:20.454 DEBUG:boto:Token: None 2024-03-04T16:29:20.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2973851a18b5425-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.456 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.456 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.456 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.456 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.456 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.457 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.457 DEBUG:boto:Data: 2024-03-04T16:29:20.457 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.457 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.457 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.457 DEBUG:boto:Params: {} 2024-03-04T16:29:20.457 DEBUG:boto:Token: None 2024-03-04T16:29:20.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE= 2024-03-04T16:29:20.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE=', 'Content-Length': '0'} 2024-03-04T16:29:20.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e7629c4a587f0c8-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.463 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.463 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.464 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.464 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.464 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.464 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.464 DEBUG:boto:Data: 2024-03-04T16:29:20.464 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.465 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.465 DEBUG:boto:Params: {} 2024-03-04T16:29:20.465 DEBUG:boto:Token: None 2024-03-04T16:29:20.465 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE= 2024-03-04T16:29:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/qNNdcTAijqnJmzVZ/ysooQafhE=', 'Content-Length': '0'} 2024-03-04T16:29:20.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8236a1a198b67b4-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.468 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.469 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.469 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.469 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.469 DEBUG:boto:Method: GET 2024-03-04T16:29:20.469 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.469 DEBUG:boto:Data: 2024-03-04T16:29:20.469 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.470 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.470 DEBUG:boto:Params: {} 2024-03-04T16:29:20.470 DEBUG:boto:Token: None 2024-03-04T16:29:20.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.471 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acfa23c2809d9c57-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.476 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.476 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.476 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.476 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.476 DEBUG:boto:Method: GET 2024-03-04T16:29:20.476 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.476 DEBUG:boto:Data: 2024-03-04T16:29:20.476 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.476 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.476 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.476 DEBUG:boto:Params: {} 2024-03-04T16:29:20.477 DEBUG:boto:Token: None 2024-03-04T16:29:20.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069f5456ea36f525e-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.491 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.491 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.491 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.491 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.491 DEBUG:boto:Method: GET 2024-03-04T16:29:20.492 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.492 DEBUG:boto:Data: 2024-03-04T16:29:20.492 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.492 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.492 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.492 DEBUG:boto:Params: {} 2024-03-04T16:29:20.492 DEBUG:boto:Token: None 2024-03-04T16:29:20.492 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.492 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000fc2a2f527a813e-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.540 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.540 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.541 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.541 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.541 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.541 DEBUG:boto:Method: GET 2024-03-04T16:29:20.541 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.541 DEBUG:boto:Data: 2024-03-04T16:29:20.541 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.541 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.541 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.541 DEBUG:boto:Params: {} 2024-03-04T16:29:20.542 DEBUG:boto:Token: None 2024-03-04T16:29:20.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8553524e2fc8d3-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.544 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.545 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.545 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.545 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.545 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.545 DEBUG:boto:Data: 2024-03-04T16:29:20.545 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.545 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.545 DEBUG:boto:Params: {} 2024-03-04T16:29:20.545 DEBUG:boto:Token: None 2024-03-04T16:29:20.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d8dee0a8bd784ae-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.548 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.548 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.548 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.548 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:29:20.549 DEBUG:boto:Data: 2024-03-04T16:29:20.549 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.549 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.549 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.549 DEBUG:boto:Params: {} 2024-03-04T16:29:20.549 DEBUG:boto:Token: None 2024-03-04T16:29:20.549 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy 2024-03-04T16:29:20.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA= 2024-03-04T16:29:20.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yete5Tub0EMvtImp565SrjDO/lA=', 'Content-Length': '0'} 2024-03-04T16:29:20.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a57240934a3b11a-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:29:20.551 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.552 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.552 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.552 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.552 DEBUG:boto:Method: GET 2024-03-04T16:29:20.552 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.552 DEBUG:boto:Data: 2024-03-04T16:29:20.552 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.552 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.552 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.552 DEBUG:boto:Params: {} 2024-03-04T16:29:20.552 DEBUG:boto:Token: None 2024-03-04T16:29:20.553 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ece3365bbe824e11-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.555 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:29:20.555 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:29:20.555 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.555 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.555 DEBUG:boto:Method: GET 2024-03-04T16:29:20.555 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.555 DEBUG:boto:Data: 2024-03-04T16:29:20.555 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.556 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.556 DEBUG:boto:Params: {} 2024-03-04T16:29:20.556 DEBUG:boto:Token: None 2024-03-04T16:29:20.556 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:29:20.556 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc= 2024-03-04T16:29:20.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6a+dHZcuIlUqdk3uIuSQDabIZwc=', 'Content-Length': '0'} 2024-03-04T16:29:20.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063da1594f6172abc-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.559 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.559 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.559 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.559 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.559 DEBUG:boto:Method: GET 2024-03-04T16:29:20.559 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.559 DEBUG:boto:Data: 2024-03-04T16:29:20.559 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.559 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.559 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.560 DEBUG:boto:Params: {} 2024-03-04T16:29:20.560 DEBUG:boto:Token: None 2024-03-04T16:29:20.560 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.560 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000364b3c0eb17bc453-0065e5f6e0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.562 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.563 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.563 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.563 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.564 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.564 DEBUG:boto:Method: GET 2024-03-04T16:29:20.564 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:29:20.564 DEBUG:boto:Data: 2024-03-04T16:29:20.564 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.564 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.564 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.564 DEBUG:boto:Params: {} 2024-03-04T16:29:20.564 DEBUG:boto:Token: None 2024-03-04T16:29:20.564 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.565 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM= 2024-03-04T16:29:20.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBKwwfUD4GiBkMUYOa2F43bfSbM=', 'Content-Length': '0'} 2024-03-04T16:29:20.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074f00b6d233dd4b2-0065e5f6e0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.612 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:29:20.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone3, test-zone2} 2024-03-04T16:29:20.613 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.613 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.613 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.613 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.613 DEBUG:boto:Data: 2024-03-04T16:29:20.613 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.613 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.614 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.614 DEBUG:boto:Params: {} 2024-03-04T16:29:20.614 DEBUG:boto:Token: None 2024-03-04T16:29:20.614 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000258a2b360e0dc969-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.617 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.617 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.618 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.618 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.618 DEBUG:boto:Data: 2024-03-04T16:29:20.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.618 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.618 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.618 DEBUG:boto:Params: {} 2024-03-04T16:29:20.618 DEBUG:boto:Token: None 2024-03-04T16:29:20.618 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.619 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-amz-request-id', 'tx000002a6b252e16fff81a-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.629 DEBUG:rgw_multi.tests:zone=test-zone1 version=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:29:20.630 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.630 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.630 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.630 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.630 DEBUG:boto:Data: 2024-03-04T16:29:20.630 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.631 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.631 DEBUG:boto:Params: {} 2024-03-04T16:29:20.631 DEBUG:boto:Token: None 2024-03-04T16:29:20.631 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.631 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000811d334cc3b34997-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.634 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.634 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.634 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.634 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.634 DEBUG:boto:Data: 2024-03-04T16:29:20.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.635 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.635 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.635 DEBUG:boto:Params: {} 2024-03-04T16:29:20.635 DEBUG:boto:Token: None 2024-03-04T16:29:20.635 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.635 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-amz-request-id', 'tx00000fa33c1004ebcde6c-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.646 DEBUG:rgw_multi.tests:zone=test-zone2 version=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:29:20.648 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.648 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.648 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.648 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.648 DEBUG:boto:Data: 2024-03-04T16:29:20.649 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.649 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.649 DEBUG:boto:Params: {} 2024-03-04T16:29:20.649 DEBUG:boto:Token: None 2024-03-04T16:29:20.649 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053317a35ddd8518a-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.653 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.653 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.653 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.653 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.654 DEBUG:boto:Data: 2024-03-04T16:29:20.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.654 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.654 DEBUG:boto:Params: {} 2024-03-04T16:29:20.654 DEBUG:boto:Token: None 2024-03-04T16:29:20.654 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-amz-request-id', 'tx00000af7570ecd0beacac-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.665 DEBUG:rgw_multi.tests:zone=test-zone3 version=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:29:20.665 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.665 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.666 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.666 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.666 DEBUG:boto:Data: 2024-03-04T16:29:20.666 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.666 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.666 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.666 DEBUG:boto:Params: {} 2024-03-04T16:29:20.667 DEBUG:boto:Token: None 2024-03-04T16:29:20.667 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.667 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b40ca51685c221-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.670 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.671 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.671 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.671 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.671 DEBUG:boto:Data: 2024-03-04T16:29:20.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.671 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.671 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.671 DEBUG:boto:Params: {} 2024-03-04T16:29:20.671 DEBUG:boto:Token: None 2024-03-04T16:29:20.672 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.672 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-amz-request-id', 'tx00000ee61c3c3ef73a94f-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.681 DEBUG:rgw_multi.tests:zone=test-zone1 version=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:29:20.681 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.681 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.681 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.681 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.681 DEBUG:boto:Data: 2024-03-04T16:29:20.681 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.681 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.681 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.682 DEBUG:boto:Params: {} 2024-03-04T16:29:20.682 DEBUG:boto:Token: None 2024-03-04T16:29:20.682 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.682 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001959de6c50cc2c84-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.684 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.685 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.685 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.685 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.685 DEBUG:boto:Data: 2024-03-04T16:29:20.686 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.686 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.686 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.686 DEBUG:boto:Params: {} 2024-03-04T16:29:20.686 DEBUG:boto:Token: None 2024-03-04T16:29:20.686 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-amz-request-id', 'tx000002bf6c58c7c6cb081-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.732 DEBUG:rgw_multi.tests:zone=test-zone2 version=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:29:20.733 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.733 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.733 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.733 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.733 DEBUG:boto:Data: 2024-03-04T16:29:20.733 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.733 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.733 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.733 DEBUG:boto:Params: {} 2024-03-04T16:29:20.734 DEBUG:boto:Token: None 2024-03-04T16:29:20.734 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.734 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c50b1fbba692e9a-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.739 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.739 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.740 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.740 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.740 DEBUG:boto:Data: 2024-03-04T16:29:20.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.740 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.740 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.740 DEBUG:boto:Params: {} 2024-03-04T16:29:20.740 DEBUG:boto:Token: None 2024-03-04T16:29:20.740 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.741 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-amz-request-id', 'tx0000031e754784d18cad1-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.795 DEBUG:rgw_multi.tests:zone=test-zone3 version=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:29:20.796 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.796 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.796 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.796 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.796 DEBUG:boto:Data: 2024-03-04T16:29:20.796 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.796 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.796 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.796 DEBUG:boto:Params: {} 2024-03-04T16:29:20.797 DEBUG:boto:Token: None 2024-03-04T16:29:20.797 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.797 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c85c75e1bb0fd5b0-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.800 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.800 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.800 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.800 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.800 DEBUG:boto:Data: 2024-03-04T16:29:20.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.801 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.801 DEBUG:boto:Params: {} 2024-03-04T16:29:20.801 DEBUG:boto:Token: None 2024-03-04T16:29:20.801 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.801 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-amz-request-id', 'tx00000aa8a8146ac332dff-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.814 DEBUG:rgw_multi.tests:zone=test-zone1 version=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:29:20.814 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.815 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.815 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.815 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.815 DEBUG:boto:Data: 2024-03-04T16:29:20.815 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.815 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.816 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.816 DEBUG:boto:Params: {} 2024-03-04T16:29:20.816 DEBUG:boto:Token: None 2024-03-04T16:29:20.816 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.816 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa44450c2718f64-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.819 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.819 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.820 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.820 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.820 DEBUG:boto:Data: 2024-03-04T16:29:20.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.820 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.820 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.820 DEBUG:boto:Params: {} 2024-03-04T16:29:20.821 DEBUG:boto:Token: None 2024-03-04T16:29:20.821 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.821 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-amz-request-id', 'tx000006fdbedf8605aae50-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.889 DEBUG:rgw_multi.tests:zone=test-zone2 version=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:29:20.890 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.890 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.890 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.890 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.890 DEBUG:boto:Data: 2024-03-04T16:29:20.890 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.890 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.890 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.890 DEBUG:boto:Params: {} 2024-03-04T16:29:20.890 DEBUG:boto:Token: None 2024-03-04T16:29:20.891 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.891 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012fdfebff6c838a1-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.893 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.893 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.894 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.894 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.894 DEBUG:boto:Data: 2024-03-04T16:29:20.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.894 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.894 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.894 DEBUG:boto:Params: {} 2024-03-04T16:29:20.895 DEBUG:boto:Token: None 2024-03-04T16:29:20.895 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.895 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-amz-request-id', 'tx00000dd9acbd94512ad2b-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.904 DEBUG:rgw_multi.tests:zone=test-zone3 version=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:29:20.904 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.904 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.904 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.904 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.904 DEBUG:boto:Data: 2024-03-04T16:29:20.904 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.905 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.905 DEBUG:boto:Params: {} 2024-03-04T16:29:20.905 DEBUG:boto:Token: None 2024-03-04T16:29:20.905 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.905 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee40cd2907bb0f5f-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.908 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.908 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.908 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.908 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.909 DEBUG:boto:Data: 2024-03-04T16:29:20.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.909 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.909 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.909 DEBUG:boto:Params: {} 2024-03-04T16:29:20.909 DEBUG:boto:Token: None 2024-03-04T16:29:20.909 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-amz-request-id', 'tx0000057aa9fa02702a66a-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.919 DEBUG:rgw_multi.tests:zone=test-zone1 version=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:29:20.920 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.920 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.920 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.920 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.920 DEBUG:boto:Data: 2024-03-04T16:29:20.920 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.920 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.921 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.921 DEBUG:boto:Params: {} 2024-03-04T16:29:20.921 DEBUG:boto:Token: None 2024-03-04T16:29:20.921 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c6e6f5ffaeb1308-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.924 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.924 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.924 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.924 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.924 DEBUG:boto:Data: 2024-03-04T16:29:20.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.924 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:20.924 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.925 DEBUG:boto:Params: {} 2024-03-04T16:29:20.925 DEBUG:boto:Token: None 2024-03-04T16:29:20.925 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.925 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-amz-request-id', 'tx00000e18dd3f546795d50-0065e5f6e0-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.935 DEBUG:rgw_multi.tests:zone=test-zone2 version=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:29:20.936 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.936 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.936 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.936 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.937 DEBUG:boto:Data: 2024-03-04T16:29:20.937 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.937 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.937 DEBUG:boto:Params: {} 2024-03-04T16:29:20.938 DEBUG:boto:Token: None 2024-03-04T16:29:20.938 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6ff81d9a550dfea-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.943 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.944 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.944 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.944 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.945 DEBUG:boto:Data: 2024-03-04T16:29:20.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:20.946 DEBUG:boto:Port: 8001 2024-03-04T16:29:20.946 DEBUG:boto:Params: {} 2024-03-04T16:29:20.946 DEBUG:boto:Token: None 2024-03-04T16:29:20.946 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:20.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-amz-request-id', 'tx000007e6318593e1ca041-0065e5f6e0-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.989 DEBUG:rgw_multi.tests:zone=test-zone3 version=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:29:20.989 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:20.989 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:20.989 DEBUG:boto:Method: HEAD 2024-03-04T16:29:20.989 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:20.989 DEBUG:boto:Data: 2024-03-04T16:29:20.989 DEBUG:boto:Headers: {} 2024-03-04T16:29:20.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.989 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.989 DEBUG:boto:Params: {} 2024-03-04T16:29:20.989 DEBUG:boto:Token: None 2024-03-04T16:29:20.990 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/ 2024-03-04T16:29:20.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg= 2024-03-04T16:29:20.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zIjip5bY1oKIgm0k58PLo4gNpEg=', 'Content-Length': '0'} 2024-03-04T16:29:20.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4eaffe32ce27b17-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:20.993 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:20.993 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:20.993 DEBUG:boto:Method: PUT 2024-03-04T16:29:20.993 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:20.993 DEBUG:boto:Data: 2024-03-04T16:29:20.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:20.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:20.993 DEBUG:boto:Port: 8000 2024-03-04T16:29:20.993 DEBUG:boto:Params: {} 2024-03-04T16:29:20.993 DEBUG:boto:Token: None 2024-03-04T16:29:20.994 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:20 GMT /sbnvju-22/obj 2024-03-04T16:29:20.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM= 2024-03-04T16:29:20.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XoTV39pYQtwLzHHcWHByPhzWtgM='} 2024-03-04T16:29:21.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-amz-request-id', 'tx00000a20637be84554bf3-0065e5f6e0-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.002 DEBUG:rgw_multi.tests:zone=test-zone1 version=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:29:21.002 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.002 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.003 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.003 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.003 DEBUG:boto:Data: 2024-03-04T16:29:21.003 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.003 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.003 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.003 DEBUG:boto:Params: {} 2024-03-04T16:29:21.003 DEBUG:boto:Token: None 2024-03-04T16:29:21.003 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.004 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f81ad7787408c2f5-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.006 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.006 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.007 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.007 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.007 DEBUG:boto:Data: 2024-03-04T16:29:21.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.007 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.007 DEBUG:boto:Params: {} 2024-03-04T16:29:21.007 DEBUG:boto:Token: None 2024-03-04T16:29:21.007 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.008 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-amz-request-id', 'tx00000d5caed30a40bcd4d-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.019 DEBUG:rgw_multi.tests:zone=test-zone2 version=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:29:21.019 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.019 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.019 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.019 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.019 DEBUG:boto:Data: 2024-03-04T16:29:21.020 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.020 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.020 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.020 DEBUG:boto:Params: {} 2024-03-04T16:29:21.020 DEBUG:boto:Token: None 2024-03-04T16:29:21.020 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a81ec8cd59a9abe4-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.023 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.023 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.023 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.023 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.024 DEBUG:boto:Data: 2024-03-04T16:29:21.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.024 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.024 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.024 DEBUG:boto:Params: {} 2024-03-04T16:29:21.025 DEBUG:boto:Token: None 2024-03-04T16:29:21.025 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-amz-request-id', 'tx000007778f7906fa8440f-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.033 DEBUG:rgw_multi.tests:zone=test-zone3 version=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:29:21.034 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.034 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.034 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.034 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.034 DEBUG:boto:Data: 2024-03-04T16:29:21.035 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.035 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.035 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.035 DEBUG:boto:Params: {} 2024-03-04T16:29:21.035 DEBUG:boto:Token: None 2024-03-04T16:29:21.035 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.035 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a562376710da97b-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.038 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.038 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.038 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.038 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.038 DEBUG:boto:Data: 2024-03-04T16:29:21.038 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.038 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.038 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.038 DEBUG:boto:Params: {} 2024-03-04T16:29:21.038 DEBUG:boto:Token: None 2024-03-04T16:29:21.039 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.039 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-amz-request-id', 'tx00000bea3631936f3829b-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.080 DEBUG:rgw_multi.tests:zone=test-zone1 version=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:29:21.081 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.081 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.081 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.081 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.081 DEBUG:boto:Data: 2024-03-04T16:29:21.081 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.081 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.081 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.081 DEBUG:boto:Params: {} 2024-03-04T16:29:21.081 DEBUG:boto:Token: None 2024-03-04T16:29:21.082 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d45706c3e9ac702-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.085 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.085 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.085 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.086 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.086 DEBUG:boto:Data: 2024-03-04T16:29:21.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.086 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.086 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.086 DEBUG:boto:Params: {} 2024-03-04T16:29:21.086 DEBUG:boto:Token: None 2024-03-04T16:29:21.087 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.088 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-amz-request-id', 'tx000008d5497ea24330426-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.098 DEBUG:rgw_multi.tests:zone=test-zone2 version=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:29:21.098 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.098 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.098 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.098 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.098 DEBUG:boto:Data: 2024-03-04T16:29:21.098 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.099 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.099 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.099 DEBUG:boto:Params: {} 2024-03-04T16:29:21.099 DEBUG:boto:Token: None 2024-03-04T16:29:21.099 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.099 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e0a17f2d92eb2e4-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.102 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.102 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.102 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.103 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.103 DEBUG:boto:Data: 2024-03-04T16:29:21.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.103 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.103 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.103 DEBUG:boto:Params: {} 2024-03-04T16:29:21.103 DEBUG:boto:Token: None 2024-03-04T16:29:21.103 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-amz-request-id', 'tx0000069f781d6c8fe0070-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.113 DEBUG:rgw_multi.tests:zone=test-zone3 version=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:29:21.113 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.113 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.114 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.114 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.114 DEBUG:boto:Data: 2024-03-04T16:29:21.114 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.114 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.114 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.114 DEBUG:boto:Params: {} 2024-03-04T16:29:21.114 DEBUG:boto:Token: None 2024-03-04T16:29:21.115 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.115 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd55b7d737dd8c41-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.118 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.118 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.118 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.119 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.119 DEBUG:boto:Data: 2024-03-04T16:29:21.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.119 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.119 DEBUG:boto:Params: {} 2024-03-04T16:29:21.119 DEBUG:boto:Token: None 2024-03-04T16:29:21.119 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-amz-request-id', 'tx00000779d727625177cb1-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.131 DEBUG:rgw_multi.tests:zone=test-zone1 version=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:29:21.131 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.131 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.131 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.131 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.131 DEBUG:boto:Data: 2024-03-04T16:29:21.132 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.132 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.132 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.132 DEBUG:boto:Params: {} 2024-03-04T16:29:21.132 DEBUG:boto:Token: None 2024-03-04T16:29:21.132 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.132 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f046d76234b530ec-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.135 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.135 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.135 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.135 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.135 DEBUG:boto:Data: 2024-03-04T16:29:21.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.136 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.136 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.136 DEBUG:boto:Params: {} 2024-03-04T16:29:21.136 DEBUG:boto:Token: None 2024-03-04T16:29:21.136 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-amz-request-id', 'tx00000f8e36d24d29b4d20-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.148 DEBUG:rgw_multi.tests:zone=test-zone2 version=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:29:21.148 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.148 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.149 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.149 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.149 DEBUG:boto:Data: 2024-03-04T16:29:21.149 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.149 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.149 DEBUG:boto:Params: {} 2024-03-04T16:29:21.149 DEBUG:boto:Token: None 2024-03-04T16:29:21.149 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000815ad6365fe1ddd5-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.195 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.195 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.196 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.196 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.196 DEBUG:boto:Data: 2024-03-04T16:29:21.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.196 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.196 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.196 DEBUG:boto:Params: {} 2024-03-04T16:29:21.196 DEBUG:boto:Token: None 2024-03-04T16:29:21.196 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.197 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-amz-request-id', 'tx00000d60d14c4916bfa59-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.246 DEBUG:rgw_multi.tests:zone=test-zone3 version=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:29:21.246 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.246 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.247 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.247 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.247 DEBUG:boto:Data: 2024-03-04T16:29:21.247 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.247 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.247 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.247 DEBUG:boto:Params: {} 2024-03-04T16:29:21.247 DEBUG:boto:Token: None 2024-03-04T16:29:21.247 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.247 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000333402446a326d1e-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.250 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.250 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.250 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.250 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.250 DEBUG:boto:Data: 2024-03-04T16:29:21.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.250 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.250 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.250 DEBUG:boto:Params: {} 2024-03-04T16:29:21.250 DEBUG:boto:Token: None 2024-03-04T16:29:21.251 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.251 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-amz-request-id', 'tx00000362b1dd9b8bde9e0-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.260 DEBUG:rgw_multi.tests:zone=test-zone1 version=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:29:21.260 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.260 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.260 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.260 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.260 DEBUG:boto:Data: 2024-03-04T16:29:21.260 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.260 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.261 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.261 DEBUG:boto:Params: {} 2024-03-04T16:29:21.261 DEBUG:boto:Token: None 2024-03-04T16:29:21.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a54753bf4694ca7-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.264 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.264 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.264 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.264 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.264 DEBUG:boto:Data: 2024-03-04T16:29:21.265 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.265 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.265 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.265 DEBUG:boto:Params: {} 2024-03-04T16:29:21.265 DEBUG:boto:Token: None 2024-03-04T16:29:21.265 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-amz-request-id', 'tx000002683884cf0c5780f-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.277 DEBUG:rgw_multi.tests:zone=test-zone2 version=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:29:21.277 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.278 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.278 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.278 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.278 DEBUG:boto:Data: 2024-03-04T16:29:21.278 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.278 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.278 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.279 DEBUG:boto:Params: {} 2024-03-04T16:29:21.279 DEBUG:boto:Token: None 2024-03-04T16:29:21.279 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.279 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223bde0e87f0421a-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.282 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.282 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.283 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.283 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.283 DEBUG:boto:Data: 2024-03-04T16:29:21.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.283 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.283 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.283 DEBUG:boto:Params: {} 2024-03-04T16:29:21.283 DEBUG:boto:Token: None 2024-03-04T16:29:21.283 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-amz-request-id', 'tx000005a4a81ab39e5a149-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.292 DEBUG:rgw_multi.tests:zone=test-zone3 version=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:29:21.292 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.292 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.293 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.293 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.293 DEBUG:boto:Data: 2024-03-04T16:29:21.293 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.293 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.293 DEBUG:boto:Params: {} 2024-03-04T16:29:21.293 DEBUG:boto:Token: None 2024-03-04T16:29:21.293 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c77198f8c842d9a-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.296 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.296 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.297 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.297 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.297 DEBUG:boto:Data: 2024-03-04T16:29:21.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.297 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.297 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.297 DEBUG:boto:Params: {} 2024-03-04T16:29:21.297 DEBUG:boto:Token: None 2024-03-04T16:29:21.297 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-amz-request-id', 'tx000004563e533ad2ae928-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.345 DEBUG:rgw_multi.tests:zone=test-zone1 version=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:29:21.345 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.345 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.345 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.345 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.345 DEBUG:boto:Data: 2024-03-04T16:29:21.346 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.346 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.346 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.346 DEBUG:boto:Params: {} 2024-03-04T16:29:21.346 DEBUG:boto:Token: None 2024-03-04T16:29:21.346 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.347 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000357148366ab37d5e-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.352 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.353 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.353 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.353 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.353 DEBUG:boto:Data: 2024-03-04T16:29:21.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.354 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.354 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.354 DEBUG:boto:Params: {} 2024-03-04T16:29:21.354 DEBUG:boto:Token: None 2024-03-04T16:29:21.355 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.355 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-amz-request-id', 'tx00000d39fa29fdc604bcf-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.401 DEBUG:rgw_multi.tests:zone=test-zone2 version=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:29:21.401 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.401 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.401 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.401 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.402 DEBUG:boto:Data: 2024-03-04T16:29:21.402 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.402 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.402 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.402 DEBUG:boto:Params: {} 2024-03-04T16:29:21.402 DEBUG:boto:Token: None 2024-03-04T16:29:21.403 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.403 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085258dfe34865989-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.406 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.406 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.406 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.406 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.406 DEBUG:boto:Data: 2024-03-04T16:29:21.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.407 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.407 DEBUG:boto:Params: {} 2024-03-04T16:29:21.407 DEBUG:boto:Token: None 2024-03-04T16:29:21.407 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-amz-request-id', 'tx00000b4ba46050faf9303-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.416 DEBUG:rgw_multi.tests:zone=test-zone3 version=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:29:21.417 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.417 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.417 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.417 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.417 DEBUG:boto:Data: 2024-03-04T16:29:21.417 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.417 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.418 DEBUG:boto:Params: {} 2024-03-04T16:29:21.418 DEBUG:boto:Token: None 2024-03-04T16:29:21.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000028a3900771fffe6-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.422 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.422 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.422 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.422 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.422 DEBUG:boto:Data: 2024-03-04T16:29:21.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.422 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:29:21.422 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.422 DEBUG:boto:Params: {} 2024-03-04T16:29:21.422 DEBUG:boto:Token: None 2024-03-04T16:29:21.423 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.423 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-amz-request-id', 'tx00000dd4b504d01e010bc-0065e5f6e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.431 DEBUG:rgw_multi.tests:zone=test-zone1 version=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:29:21.431 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.431 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.431 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.432 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.432 DEBUG:boto:Data: 2024-03-04T16:29:21.432 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.432 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.432 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.432 DEBUG:boto:Params: {} 2024-03-04T16:29:21.432 DEBUG:boto:Token: None 2024-03-04T16:29:21.432 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af6aca2a5a54f0e8-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.436 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.436 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.436 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.436 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.436 DEBUG:boto:Data: 2024-03-04T16:29:21.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:29:21.437 DEBUG:boto:Port: 8000 2024-03-04T16:29:21.437 DEBUG:boto:Params: {} 2024-03-04T16:29:21.437 DEBUG:boto:Token: None 2024-03-04T16:29:21.437 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-amz-request-id', 'tx00000181abc5ca61e6442-0065e5f6e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.448 DEBUG:rgw_multi.tests:zone=test-zone2 version=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:29:21.448 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:29:21.448 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:29:21.449 DEBUG:boto:Method: HEAD 2024-03-04T16:29:21.449 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:29:21.449 DEBUG:boto:Data: 2024-03-04T16:29:21.449 DEBUG:boto:Headers: {} 2024-03-04T16:29:21.449 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.449 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.449 DEBUG:boto:Params: {} 2024-03-04T16:29:21.449 DEBUG:boto:Token: None 2024-03-04T16:29:21.449 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/ 2024-03-04T16:29:21.449 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw= 2024-03-04T16:29:21.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbM4RgjFiIkbWB+9oTh63trbdSw=', 'Content-Length': '0'} 2024-03-04T16:29:21.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017cbbc56a2455035-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.452 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:29:21.452 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:29:21.452 DEBUG:boto:Method: PUT 2024-03-04T16:29:21.452 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:29:21.453 DEBUG:boto:Data: 2024-03-04T16:29:21.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-03-04T16:29:21.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:29:21.453 DEBUG:boto:Port: 8001 2024-03-04T16:29:21.453 DEBUG:boto:Params: {} 2024-03-04T16:29:21.453 DEBUG:boto:Token: None 2024-03-04T16:29:21.453 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 04 Mar 2024 16:29:21 GMT /sbnvju-22/obj 2024-03-04T16:29:21.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s= 2024-03-04T16:29:21.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:29:21 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KJ7aS/6OdpbWNy0E6S9d9KIUp2s='} 2024-03-04T16:29:21.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-amz-request-id', 'tx0000041f7dd7967fc3a3c-0065e5f6e1-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:29:21.497 DEBUG:rgw_multi.tests:zone=test-zone3 version=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:29:21.497 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-22 2024-03-04T16:29:21.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:21.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:21.497 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:29:21.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:29:21.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:29:21.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:21.609+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:21.609 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , , , ] 2024-03-04T16:29:21.609 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:26.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:26.611+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:26.611 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , , , ] 2024-03-04T16:29:26.611 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:31.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:31.612+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:31.612 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , , , ] 2024-03-04T16:29:31.612 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:36.614+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:36.614 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000015.52.6, , ] 2024-03-04T16:29:36.614 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:41.615+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:41.615 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000015.52.6, , ] 2024-03-04T16:29:41.615 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:46.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:46.616+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:46.616 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000015.52.6, , ] 2024-03-04T16:29:46.616 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:51.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:51.618+0000 7f582a667800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:51.617 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000015.52.6, , ] 2024-03-04T16:29:51.617 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:56.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:56.619+0000 7f582a667800 1 bucket sync caught up with source: 2024-03-04T16:29:56.618 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000040.116.6, , ] 2024-03-04T16:29:56.618 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000020.61.12, , ] 2024-03-04T16:29:56.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:29:56.619+0000 7f582a667800 0 bucket checkpoint complete 2024-03-04T16:29:56.641 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-22 2024-03-04T16:29:56.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:56.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:29:56.642 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:29:56.704 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:29:56.704 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:29:56.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:29:56.762+0000 7fe130726800 1 waiting for incremental sync to catch up: 2024-03-04T16:29:56.761 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000033.100.6, , ] 2024-03-04T16:29:56.761 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000040.116.6, , ] 2024-03-04T16:30:01.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:01.764+0000 7fe130726800 1 bucket sync caught up with source: 2024-03-04T16:30:01.763 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000005.30.10, , , 00000000040.116.6, , ] 2024-03-04T16:30:01.763 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000005.30.10, , , 00000000040.116.6, , ] 2024-03-04T16:30:01.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:01.764+0000 7fe130726800 0 bucket checkpoint complete 2024-03-04T16:30:01.786 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-22 2024-03-04T16:30:01.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:30:01.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:30:01.787 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:30:01.846 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:30:01.846 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:30:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:01.901+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:01.900 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000013.50.6, , ] 2024-03-04T16:30:01.900 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:06.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:06.903+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:06.902 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000013.50.6, , ] 2024-03-04T16:30:06.902 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:11.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:11.905+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:11.903 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000013.50.6, , ] 2024-03-04T16:30:11.903 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:16.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:16.906+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:16.904 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000015.54.6, , ] 2024-03-04T16:30:16.904 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:21.907+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:21.906 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000015.54.6, , ] 2024-03-04T16:30:21.906 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:26.909+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:26.907 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000015.54.6, , ] 2024-03-04T16:30:26.907 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:31.910+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:31.908 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000015.54.6, , ] 2024-03-04T16:30:31.908 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:36.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:36.911+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:36.909 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:30:36.909 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:41.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:41.913+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:41.910 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:30:41.910 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:46.914+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:46.911 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:30:46.912 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:51.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:51.915+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:51.913 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:30:51.913 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:30:56.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:30:56.917+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:30:56.914 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:30:56.914 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:01.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:01.918+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:01.915 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:01.915 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:06.920+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:06.919 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:06.919 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:11.921+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:11.918 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:11.918 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:16.922+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:16.919 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:16.919 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:21.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:21.924+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:21.920 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:21.920 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:26.925+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:26.921 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:26.921 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:31.927+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:31.923 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:31.923 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:36.928+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:36.924 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:36.924 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:41.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:41.930+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:41.925 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:41.925 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:46.931+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:46.926 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:46.927 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:51.932+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:51.928 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:51.928 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:31:56.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:31:56.934+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:31:56.929 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:31:56.929 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:01.936+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:01.931 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:01.932 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:06.937+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:06.933 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:06.933 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:11.939+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:16.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:16.940+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:16.935 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:16.935 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:21.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:21.942+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:21.936 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:21.936 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:26.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:26.943+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:26.938 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:26.938 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:31.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:31.944+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:31.939 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000020.63.12, , ] 2024-03-04T16:32:31.939 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:36.946+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:36.940 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:32:36.940 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:41.947+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:41.941 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:32:41.941 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:46.948+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:46.943 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:32:46.943 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:51.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:51.950+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:51.944 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:32:51.944 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:32:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:32:56.951+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:32:56.945 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:32:56.945 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:01.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:01.953+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:01.946 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:01.946 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:06.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:06.954+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:06.948 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:06.948 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:11.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:11.956+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:11.949 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:11.949 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:16.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:16.957+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:16.950 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:16.950 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:21.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:21.958+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:21.951 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:21.952 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:26.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:26.960+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:26.953 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:26.953 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:31.961+0000 7f99c4a13800 1 waiting for incremental sync to catch up: 2024-03-04T16:33:31.954 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000022.69.6, , ] 2024-03-04T16:33:31.954 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:36.963+0000 7f99c4a13800 1 bucket sync caught up with source: 2024-03-04T16:33:36.956 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:36.956 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:36.963+0000 7f99c4a13800 0 bucket checkpoint complete 2024-03-04T16:33:36.979 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-22 2024-03-04T16:33:36.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:36.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:36.981 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:33:37.039 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:37.039 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:37.102+0000 7faf46111800 1 bucket sync caught up with source: 2024-03-04T16:33:37.094 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:37.094 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000003.30.10, , , 00000000044.129.10, , ] 2024-03-04T16:33:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:37.102+0000 7faf46111800 0 bucket checkpoint complete 2024-03-04T16:33:37.113 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-22 2024-03-04T16:33:37.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:37.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:37.114 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:33:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:37.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:37.229+0000 7fea8f51f800 1 bucket sync caught up with source: 2024-03-04T16:33:37.221 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , 00000000004.35.10, , , 00000000044.130.10, , ] 2024-03-04T16:33:37.221 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , 00000000004.35.10, , , 00000000044.130.10, , ] 2024-03-04T16:33:37.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:33:37.229+0000 7fea8f51f800 0 bucket checkpoint complete 2024-03-04T16:33:37.240 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-22 2024-03-04T16:33:37.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:37.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:33:37.240 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:33:37.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:37.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:37.368 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:33:37.375+0000 7f8094f3c800 1 bucket sync caught up with source: 2024-03-04T16:33:37.368 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , 00000000004.35.10, , , 00000000044.130.10, , ] 2024-03-04T16:33:37.368 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , 00000000004.35.10, , , 00000000044.130.10, , ] 2024-03-04T16:33:37.368 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:33:37.375+0000 7f8094f3c800 0 bucket checkpoint complete 2024-03-04T16:33:37.394 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone2, test-zone1} 2024-03-04T16:33:37.394 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.394 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.394 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.394 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:37.394 DEBUG:boto:Data: 2024-03-04T16:33:37.394 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.395 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.395 DEBUG:boto:Params: {} 2024-03-04T16:33:37.395 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:33:37.396 DEBUG:boto:Token: None 2024-03-04T16:33:37.396 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.396 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MB+5J7M3J0R9gd6Qb2u8X7EoVLw= 2024-03-04T16:33:37.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MB+5J7M3J0R9gd6Qb2u8X7EoVLw=', 'Content-Length': '0'} 2024-03-04T16:33:37.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a778c3e23527fe4c-0065e5f7e1-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.401 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.401 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.401 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.402 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:37.402 DEBUG:boto:Data: 2024-03-04T16:33:37.402 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.402 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.402 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.402 DEBUG:boto:Params: {} 2024-03-04T16:33:37.402 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:33:37.402 DEBUG:boto:Token: None 2024-03-04T16:33:37.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.403 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MB+5J7M3J0R9gd6Qb2u8X7EoVLw= 2024-03-04T16:33:37.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MB+5J7M3J0R9gd6Qb2u8X7EoVLw=', 'Content-Length': '0'} 2024-03-04T16:33:37.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077e86c2a188a8560-0065e5f7e1-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.406 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:33:37.406 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.407 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.407 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.407 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.407 DEBUG:boto:Method: GET 2024-03-04T16:33:37.407 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.407 DEBUG:boto:Data: 2024-03-04T16:33:37.407 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.407 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.407 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.407 DEBUG:boto:Params: {} 2024-03-04T16:33:37.407 DEBUG:boto:Token: None 2024-03-04T16:33:37.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/?versions 2024-03-04T16:33:37.408 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY= 2024-03-04T16:33:37.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c4396e5e1f402a-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.411 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.416 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:33:37.416 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.416 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.416 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.416 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.416 DEBUG:boto:Method: GET 2024-03-04T16:33:37.416 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.416 DEBUG:boto:Data: 2024-03-04T16:33:37.416 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.417 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.417 DEBUG:boto:Params: {} 2024-03-04T16:33:37.417 DEBUG:boto:Token: None 2024-03-04T16:33:37.417 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/?versions 2024-03-04T16:33:37.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY= 2024-03-04T16:33:37.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff8bef0a7a00b4d-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.420 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.421 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:37.424 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.424 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.424 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.424 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.425 DEBUG:boto:Method: GET 2024-03-04T16:33:37.425 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.425 DEBUG:boto:Data: 2024-03-04T16:33:37.425 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.425 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.425 DEBUG:boto:Params: {} 2024-03-04T16:33:37.425 DEBUG:boto:Token: None 2024-03-04T16:33:37.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/?versions 2024-03-04T16:33:37.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY= 2024-03-04T16:33:37.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91e0fec24da0f0f-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.428 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.430 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.430 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.430 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.430 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.430 DEBUG:boto:Method: GET 2024-03-04T16:33:37.430 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:37.431 DEBUG:boto:Data: 2024-03-04T16:33:37.431 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.431 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.431 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.431 DEBUG:boto:Params: {} 2024-03-04T16:33:37.431 DEBUG:boto:Token: None 2024-03-04T16:33:37.431 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/?versions 2024-03-04T16:33:37.431 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY= 2024-03-04T16:33:37.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n49UATTucqI6WlhifSC+waHbuXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a40c31c7a9022fa0-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.434 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:37.436 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.436 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.436 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.436 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.436 DEBUG:boto:Method: GET 2024-03-04T16:33:37.437 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.437 DEBUG:boto:Data: 2024-03-04T16:33:37.437 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.437 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.437 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.437 DEBUG:boto:Params: {} 2024-03-04T16:33:37.437 DEBUG:boto:Token: None 2024-03-04T16:33:37.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2aed03c9b19eb75-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.440 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.440 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.440 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.440 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.440 DEBUG:boto:Method: GET 2024-03-04T16:33:37.440 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.440 DEBUG:boto:Data: 2024-03-04T16:33:37.440 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.440 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.440 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.441 DEBUG:boto:Params: {} 2024-03-04T16:33:37.441 DEBUG:boto:Token: None 2024-03-04T16:33:37.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9124df1fd30bcb0-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.443 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.443 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.443 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.443 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.443 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.443 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.443 DEBUG:boto:Data: 2024-03-04T16:33:37.443 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.443 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.443 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.444 DEBUG:boto:Params: {} 2024-03-04T16:33:37.444 DEBUG:boto:Token: None 2024-03-04T16:33:37.444 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.444 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uJBXrls51segcoTvveVqo19ohl0= 2024-03-04T16:33:37.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uJBXrls51segcoTvveVqo19ohl0=', 'Content-Length': '0'} 2024-03-04T16:33:37.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b19b1ee7b7bde7b-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.446 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.446 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.446 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.446 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.446 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.446 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.446 DEBUG:boto:Data: 2024-03-04T16:33:37.447 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.447 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.447 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.447 DEBUG:boto:Params: {} 2024-03-04T16:33:37.447 DEBUG:boto:Token: None 2024-03-04T16:33:37.447 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uJBXrls51segcoTvveVqo19ohl0= 2024-03-04T16:33:37.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uJBXrls51segcoTvveVqo19ohl0=', 'Content-Length': '0'} 2024-03-04T16:33:37.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f1edea84adcf7c7-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:37.449 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.449 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.449 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.449 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.449 DEBUG:boto:Method: GET 2024-03-04T16:33:37.450 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.450 DEBUG:boto:Data: 2024-03-04T16:33:37.450 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.450 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.450 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.450 DEBUG:boto:Params: {} 2024-03-04T16:33:37.450 DEBUG:boto:Token: None 2024-03-04T16:33:37.450 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3c2a2226f3dff95-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.452 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.452 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.452 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.452 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.452 DEBUG:boto:Method: GET 2024-03-04T16:33:37.453 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.453 DEBUG:boto:Data: 2024-03-04T16:33:37.453 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.453 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.453 DEBUG:boto:Params: {} 2024-03-04T16:33:37.453 DEBUG:boto:Token: None 2024-03-04T16:33:37.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e4dd5b89edd7d0c8-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.455 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.456 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.456 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.456 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.456 DEBUG:boto:Method: GET 2024-03-04T16:33:37.456 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.456 DEBUG:boto:Data: 2024-03-04T16:33:37.456 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.456 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.456 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.456 DEBUG:boto:Params: {} 2024-03-04T16:33:37.457 DEBUG:boto:Token: None 2024-03-04T16:33:37.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db32f4716fd42ea3-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.459 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.460 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.460 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.460 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.460 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.460 DEBUG:boto:Method: GET 2024-03-04T16:33:37.460 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.460 DEBUG:boto:Data: 2024-03-04T16:33:37.460 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.460 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.461 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.461 DEBUG:boto:Params: {} 2024-03-04T16:33:37.461 DEBUG:boto:Token: None 2024-03-04T16:33:37.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c6f34301fe58ed-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.508 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.508 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.509 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.509 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.509 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:37.509 DEBUG:boto:Data: 2024-03-04T16:33:37.509 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.509 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.509 DEBUG:boto:Params: {} 2024-03-04T16:33:37.509 DEBUG:boto:Token: None 2024-03-04T16:33:37.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy 2024-03-04T16:33:37.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zZhpzje1m4hULNrG3i8cvH7chNs= 2024-03-04T16:33:37.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zZhpzje1m4hULNrG3i8cvH7chNs=', 'Content-Length': '0'} 2024-03-04T16:33:37.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003cd472fa4c427c7-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.512 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.512 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.512 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.512 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:37.512 DEBUG:boto:Data: 2024-03-04T16:33:37.512 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.513 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.513 DEBUG:boto:Params: {} 2024-03-04T16:33:37.513 DEBUG:boto:Token: None 2024-03-04T16:33:37.513 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy 2024-03-04T16:33:37.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zZhpzje1m4hULNrG3i8cvH7chNs= 2024-03-04T16:33:37.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zZhpzje1m4hULNrG3i8cvH7chNs=', 'Content-Length': '0'} 2024-03-04T16:33:37.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b1a9aeed2f9217a-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:37.515 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.516 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.516 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.516 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.516 DEBUG:boto:Method: GET 2024-03-04T16:33:37.516 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.516 DEBUG:boto:Data: 2024-03-04T16:33:37.516 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.516 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.516 DEBUG:boto:Params: {} 2024-03-04T16:33:37.516 DEBUG:boto:Token: None 2024-03-04T16:33:37.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4ec83dbd3db7ef5-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.519 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:37.519 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:37.519 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.519 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.519 DEBUG:boto:Method: GET 2024-03-04T16:33:37.519 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.519 DEBUG:boto:Data: 2024-03-04T16:33:37.519 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.519 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.519 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.520 DEBUG:boto:Params: {} 2024-03-04T16:33:37.520 DEBUG:boto:Token: None 2024-03-04T16:33:37.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:37.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI= 2024-03-04T16:33:37.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8iEWG+mcJnOZaEZMmPBNMPDIYpI=', 'Content-Length': '0'} 2024-03-04T16:33:37.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b2ba6ea11084abf-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.522 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.522 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.522 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.523 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.523 DEBUG:boto:Method: GET 2024-03-04T16:33:37.523 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.523 DEBUG:boto:Data: 2024-03-04T16:33:37.523 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.524 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.524 DEBUG:boto:Params: {} 2024-03-04T16:33:37.524 DEBUG:boto:Token: None 2024-03-04T16:33:37.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdec73fde5b85804-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.528 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.528 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.528 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.528 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.529 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.529 DEBUG:boto:Method: GET 2024-03-04T16:33:37.529 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:37.529 DEBUG:boto:Data: 2024-03-04T16:33:37.529 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.529 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.529 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.529 DEBUG:boto:Params: {} 2024-03-04T16:33:37.530 DEBUG:boto:Token: None 2024-03-04T16:33:37.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b5b742a14fa3f5-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.533 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.533 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.533 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.533 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.534 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.534 DEBUG:boto:Method: GET 2024-03-04T16:33:37.534 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.534 DEBUG:boto:Data: 2024-03-04T16:33:37.534 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.534 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.534 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.534 DEBUG:boto:Params: {} 2024-03-04T16:33:37.535 DEBUG:boto:Token: None 2024-03-04T16:33:37.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.535 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d9475a965200f13-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.537 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.538 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.538 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.538 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.538 DEBUG:boto:Method: GET 2024-03-04T16:33:37.538 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.538 DEBUG:boto:Data: 2024-03-04T16:33:37.538 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.538 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.539 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.539 DEBUG:boto:Params: {} 2024-03-04T16:33:37.539 DEBUG:boto:Token: None 2024-03-04T16:33:37.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.540 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000395d17cb3816c176-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.542 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.542 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.542 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.542 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.543 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.543 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.543 DEBUG:boto:Data: 2024-03-04T16:33:37.543 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.543 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.543 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.543 DEBUG:boto:Params: {} 2024-03-04T16:33:37.544 DEBUG:boto:Token: None 2024-03-04T16:33:37.544 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.544 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3f354Ov9xk5W3GXgfIdU49pCDo= 2024-03-04T16:33:37.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3f354Ov9xk5W3GXgfIdU49pCDo=', 'Content-Length': '0'} 2024-03-04T16:33:37.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be0d7bc5d63ee95e-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.546 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.546 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.546 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.547 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.547 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.547 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.547 DEBUG:boto:Data: 2024-03-04T16:33:37.547 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.547 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.547 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.547 DEBUG:boto:Params: {} 2024-03-04T16:33:37.548 DEBUG:boto:Token: None 2024-03-04T16:33:37.548 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u3f354Ov9xk5W3GXgfIdU49pCDo= 2024-03-04T16:33:37.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u3f354Ov9xk5W3GXgfIdU49pCDo=', 'Content-Length': '0'} 2024-03-04T16:33:37.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000437f0d6186940e83-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.551 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.551 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.551 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.551 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.551 DEBUG:boto:Method: GET 2024-03-04T16:33:37.551 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.551 DEBUG:boto:Data: 2024-03-04T16:33:37.552 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.552 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.552 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.552 DEBUG:boto:Params: {} 2024-03-04T16:33:37.552 DEBUG:boto:Token: None 2024-03-04T16:33:37.552 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e5690fa31378faf-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.600 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.600 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.600 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.600 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.601 DEBUG:boto:Method: GET 2024-03-04T16:33:37.601 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.601 DEBUG:boto:Data: 2024-03-04T16:33:37.601 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.601 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.601 DEBUG:boto:Params: {} 2024-03-04T16:33:37.602 DEBUG:boto:Token: None 2024-03-04T16:33:37.602 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021f0cc4d08271b5e-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.644 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.644 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.644 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.644 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.645 DEBUG:boto:Method: GET 2024-03-04T16:33:37.645 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.645 DEBUG:boto:Data: 2024-03-04T16:33:37.645 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.645 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.645 DEBUG:boto:Params: {} 2024-03-04T16:33:37.645 DEBUG:boto:Token: None 2024-03-04T16:33:37.645 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc6bb84e5676e4c6-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.649 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.649 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.649 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.649 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.649 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.649 DEBUG:boto:Method: GET 2024-03-04T16:33:37.650 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.650 DEBUG:boto:Data: 2024-03-04T16:33:37.650 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.650 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.650 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.650 DEBUG:boto:Params: {} 2024-03-04T16:33:37.650 DEBUG:boto:Token: None 2024-03-04T16:33:37.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff4fd19ba56f0f3-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.696 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.697 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.697 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.697 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.697 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.697 DEBUG:boto:Data: 2024-03-04T16:33:37.697 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.697 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.697 DEBUG:boto:Params: {} 2024-03-04T16:33:37.697 DEBUG:boto:Token: None 2024-03-04T16:33:37.698 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd110cef608f1ada-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.701 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.701 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.701 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.701 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.701 DEBUG:boto:Data: 2024-03-04T16:33:37.701 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.701 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.701 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.701 DEBUG:boto:Params: {} 2024-03-04T16:33:37.702 DEBUG:boto:Token: None 2024-03-04T16:33:37.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.702 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e229a8e2b185bb9-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.705 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.705 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.706 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.706 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.706 DEBUG:boto:Method: GET 2024-03-04T16:33:37.706 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.706 DEBUG:boto:Data: 2024-03-04T16:33:37.706 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.706 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.706 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.706 DEBUG:boto:Params: {} 2024-03-04T16:33:37.706 DEBUG:boto:Token: None 2024-03-04T16:33:37.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.707 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b13381391406486d-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.709 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.709 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.709 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.709 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.709 DEBUG:boto:Method: GET 2024-03-04T16:33:37.709 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.710 DEBUG:boto:Data: 2024-03-04T16:33:37.710 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.710 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.710 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.710 DEBUG:boto:Params: {} 2024-03-04T16:33:37.710 DEBUG:boto:Token: None 2024-03-04T16:33:37.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f637842d94d6afd-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.712 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.712 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.712 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.713 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.713 DEBUG:boto:Method: GET 2024-03-04T16:33:37.713 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.713 DEBUG:boto:Data: 2024-03-04T16:33:37.713 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.713 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.713 DEBUG:boto:Params: {} 2024-03-04T16:33:37.713 DEBUG:boto:Token: None 2024-03-04T16:33:37.713 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003acdfa20668d03f-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.716 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.716 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.717 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.717 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.717 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.717 DEBUG:boto:Method: GET 2024-03-04T16:33:37.717 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.717 DEBUG:boto:Data: 2024-03-04T16:33:37.717 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.717 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.717 DEBUG:boto:Params: {} 2024-03-04T16:33:37.717 DEBUG:boto:Token: None 2024-03-04T16:33:37.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004229b24ef64d00a6-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.768 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.769 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.769 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.769 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.769 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.769 DEBUG:boto:Method: GET 2024-03-04T16:33:37.769 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.770 DEBUG:boto:Data: 2024-03-04T16:33:37.770 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.770 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.770 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.770 DEBUG:boto:Params: {} 2024-03-04T16:33:37.770 DEBUG:boto:Token: None 2024-03-04T16:33:37.770 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.770 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y= 2024-03-04T16:33:37.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y=', 'Content-Length': '0'} 2024-03-04T16:33:37.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdd3c36c3c3ad95c-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.772 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.772 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.773 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.773 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.773 DEBUG:boto:Method: GET 2024-03-04T16:33:37.773 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.773 DEBUG:boto:Data: 2024-03-04T16:33:37.773 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.773 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.773 DEBUG:boto:Params: {} 2024-03-04T16:33:37.773 DEBUG:boto:Token: None 2024-03-04T16:33:37.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y= 2024-03-04T16:33:37.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y=', 'Content-Length': '0'} 2024-03-04T16:33:37.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b7c6df0be3a88eb-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.776 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.776 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.776 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.776 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.776 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.776 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.776 DEBUG:boto:Data: 2024-03-04T16:33:37.777 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.777 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.777 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.777 DEBUG:boto:Params: {} 2024-03-04T16:33:37.777 DEBUG:boto:Token: None 2024-03-04T16:33:37.777 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bnCz1/cord3AoFH4z3mM0V4RywU= 2024-03-04T16:33:37.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bnCz1/cord3AoFH4z3mM0V4RywU=', 'Content-Length': '0'} 2024-03-04T16:33:37.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb45f3fe6e1796f1-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.779 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.780 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.780 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.780 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.780 DEBUG:boto:Data: 2024-03-04T16:33:37.780 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.780 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.780 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.780 DEBUG:boto:Params: {} 2024-03-04T16:33:37.780 DEBUG:boto:Token: None 2024-03-04T16:33:37.781 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.781 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bnCz1/cord3AoFH4z3mM0V4RywU= 2024-03-04T16:33:37.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bnCz1/cord3AoFH4z3mM0V4RywU=', 'Content-Length': '0'} 2024-03-04T16:33:37.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f532b5c5d3b0aae1-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.783 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.783 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.783 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.783 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.783 DEBUG:boto:Method: GET 2024-03-04T16:33:37.783 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.784 DEBUG:boto:Data: 2024-03-04T16:33:37.784 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.784 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.784 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.784 DEBUG:boto:Params: {} 2024-03-04T16:33:37.784 DEBUG:boto:Token: None 2024-03-04T16:33:37.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.784 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y= 2024-03-04T16:33:37.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y=', 'Content-Length': '0'} 2024-03-04T16:33:37.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005279e91e39895983-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.828 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.828 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.828 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.829 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.829 DEBUG:boto:Method: GET 2024-03-04T16:33:37.829 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.829 DEBUG:boto:Data: 2024-03-04T16:33:37.829 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.829 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.829 DEBUG:boto:Params: {} 2024-03-04T16:33:37.829 DEBUG:boto:Token: None 2024-03-04T16:33:37.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:37.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y= 2024-03-04T16:33:37.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x9eSLm28zUUFIRlj84IQFi64I7Y=', 'Content-Length': '0'} 2024-03-04T16:33:37.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000665610601f438cf6-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.832 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.832 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.832 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.832 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.832 DEBUG:boto:Method: GET 2024-03-04T16:33:37.832 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.832 DEBUG:boto:Data: 2024-03-04T16:33:37.833 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.833 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.833 DEBUG:boto:Params: {} 2024-03-04T16:33:37.833 DEBUG:boto:Token: None 2024-03-04T16:33:37.833 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee61540e36293cb2-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.836 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.836 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.836 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.836 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.836 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.836 DEBUG:boto:Method: GET 2024-03-04T16:33:37.837 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.837 DEBUG:boto:Data: 2024-03-04T16:33:37.837 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.837 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.837 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.837 DEBUG:boto:Params: {} 2024-03-04T16:33:37.837 DEBUG:boto:Token: None 2024-03-04T16:33:37.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f561e4c806638fe-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.840 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.840 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.840 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.841 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.841 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.841 DEBUG:boto:Data: 2024-03-04T16:33:37.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.841 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.841 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.841 DEBUG:boto:Params: {} 2024-03-04T16:33:37.841 DEBUG:boto:Token: None 2024-03-04T16:33:37.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.841 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e5e756abd642324-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.844 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.844 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.845 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.845 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.845 DEBUG:boto:Data: 2024-03-04T16:33:37.845 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.845 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.845 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.845 DEBUG:boto:Params: {} 2024-03-04T16:33:37.845 DEBUG:boto:Token: None 2024-03-04T16:33:37.845 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcfe8e596470c803-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.849 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.849 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.849 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.849 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.849 DEBUG:boto:Method: GET 2024-03-04T16:33:37.849 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.849 DEBUG:boto:Data: 2024-03-04T16:33:37.849 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.850 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.850 DEBUG:boto:Params: {} 2024-03-04T16:33:37.850 DEBUG:boto:Token: None 2024-03-04T16:33:37.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d11d45336ce2c25f-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.852 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.852 DEBUG:boto:Method: GET 2024-03-04T16:33:37.852 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.853 DEBUG:boto:Data: 2024-03-04T16:33:37.853 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.853 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.853 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.853 DEBUG:boto:Params: {} 2024-03-04T16:33:37.853 DEBUG:boto:Token: None 2024-03-04T16:33:37.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7bfa58a8f1f7962-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.900 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.900 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.900 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.900 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.901 DEBUG:boto:Method: GET 2024-03-04T16:33:37.901 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.901 DEBUG:boto:Data: 2024-03-04T16:33:37.901 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.901 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.901 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.901 DEBUG:boto:Params: {} 2024-03-04T16:33:37.901 DEBUG:boto:Token: None 2024-03-04T16:33:37.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.902 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0fd49fc26170fc2-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.904 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.904 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.905 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.905 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.905 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.905 DEBUG:boto:Method: GET 2024-03-04T16:33:37.905 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.905 DEBUG:boto:Data: 2024-03-04T16:33:37.905 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.905 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.905 DEBUG:boto:Params: {} 2024-03-04T16:33:37.906 DEBUG:boto:Token: None 2024-03-04T16:33:37.906 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.906 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071f3d28235351cb4-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.908 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.909 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.909 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.909 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.909 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.909 DEBUG:boto:Method: GET 2024-03-04T16:33:37.909 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.909 DEBUG:boto:Data: 2024-03-04T16:33:37.909 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.909 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.910 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.910 DEBUG:boto:Params: {} 2024-03-04T16:33:37.910 DEBUG:boto:Token: None 2024-03-04T16:33:37.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI= 2024-03-04T16:33:37.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI=', 'Content-Length': '0'} 2024-03-04T16:33:37.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d3389a690570159-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.912 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.912 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.912 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.912 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.913 DEBUG:boto:Method: GET 2024-03-04T16:33:37.913 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.913 DEBUG:boto:Data: 2024-03-04T16:33:37.913 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.913 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.913 DEBUG:boto:Params: {} 2024-03-04T16:33:37.913 DEBUG:boto:Token: None 2024-03-04T16:33:37.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI= 2024-03-04T16:33:37.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI=', 'Content-Length': '0'} 2024-03-04T16:33:37.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035d7755a5155f62f-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.915 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.915 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.916 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.916 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.916 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.916 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.916 DEBUG:boto:Data: 2024-03-04T16:33:37.916 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.916 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.916 DEBUG:boto:Params: {} 2024-03-04T16:33:37.916 DEBUG:boto:Token: None 2024-03-04T16:33:37.917 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.917 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QEOrtm8Qu/zkA9o7hEVws7CgsXY= 2024-03-04T16:33:37.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QEOrtm8Qu/zkA9o7hEVws7CgsXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e714a3ec5bf1aae-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.918 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.919 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.919 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.919 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.919 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.919 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.919 DEBUG:boto:Data: 2024-03-04T16:33:37.919 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.919 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.919 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.919 DEBUG:boto:Params: {} 2024-03-04T16:33:37.919 DEBUG:boto:Token: None 2024-03-04T16:33:37.920 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.920 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QEOrtm8Qu/zkA9o7hEVws7CgsXY= 2024-03-04T16:33:37.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QEOrtm8Qu/zkA9o7hEVws7CgsXY=', 'Content-Length': '0'} 2024-03-04T16:33:37.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022aff1458f52f577-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.922 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.922 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.922 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.922 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.922 DEBUG:boto:Method: GET 2024-03-04T16:33:37.922 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.922 DEBUG:boto:Data: 2024-03-04T16:33:37.923 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.923 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.923 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.923 DEBUG:boto:Params: {} 2024-03-04T16:33:37.923 DEBUG:boto:Token: None 2024-03-04T16:33:37.923 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.923 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI= 2024-03-04T16:33:37.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI=', 'Content-Length': '0'} 2024-03-04T16:33:37.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042ecb369f83c65e2-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.925 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.925 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.926 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.926 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.926 DEBUG:boto:Method: GET 2024-03-04T16:33:37.926 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.926 DEBUG:boto:Data: 2024-03-04T16:33:37.926 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.926 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.926 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.926 DEBUG:boto:Params: {} 2024-03-04T16:33:37.926 DEBUG:boto:Token: None 2024-03-04T16:33:37.926 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:37.927 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI= 2024-03-04T16:33:37.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IKuSDAYUJXL99iHJJVXD3AxGaGI=', 'Content-Length': '0'} 2024-03-04T16:33:37.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f673130a20056dd1-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.928 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.929 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.929 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.929 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.929 DEBUG:boto:Method: GET 2024-03-04T16:33:37.929 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.929 DEBUG:boto:Data: 2024-03-04T16:33:37.929 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.929 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.929 DEBUG:boto:Params: {} 2024-03-04T16:33:37.929 DEBUG:boto:Token: None 2024-03-04T16:33:37.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edeff824935c2e3f-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.972 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.973 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:37.973 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:37.973 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.973 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:37.973 DEBUG:boto:Method: GET 2024-03-04T16:33:37.973 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:37.973 DEBUG:boto:Data: 2024-03-04T16:33:37.973 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.973 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.973 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.973 DEBUG:boto:Params: {} 2024-03-04T16:33:37.974 DEBUG:boto:Token: None 2024-03-04T16:33:37.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/ 2024-03-04T16:33:37.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8= 2024-03-04T16:33:37.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6H0XAA+VgygQ4yzsDFQcCHAPPv8=', 'Content-Length': '0'} 2024-03-04T16:33:37.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8cf31497236793-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.977 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:37.977 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.977 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.977 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.977 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.977 DEBUG:boto:Data: 2024-03-04T16:33:37.977 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.977 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.977 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.978 DEBUG:boto:Params: {} 2024-03-04T16:33:37.978 DEBUG:boto:Token: None 2024-03-04T16:33:37.978 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d383e861a430121-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.981 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.981 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.982 DEBUG:boto:Method: HEAD 2024-03-04T16:33:37.982 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:37.982 DEBUG:boto:Data: 2024-03-04T16:33:37.982 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.982 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.982 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.982 DEBUG:boto:Params: {} 2024-03-04T16:33:37.982 DEBUG:boto:Token: None 2024-03-04T16:33:37.982 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj 2024-03-04T16:33:37.982 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc= 2024-03-04T16:33:37.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Taju/U3/KbARsHpxh8hYEc7Lwrc=', 'Content-Length': '0'} 2024-03-04T16:33:37.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edd31459c2717473-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:37.986 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.986 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.986 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.986 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.986 DEBUG:boto:Method: GET 2024-03-04T16:33:37.986 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.986 DEBUG:boto:Data: 2024-03-04T16:33:37.986 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.986 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:37.986 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.987 DEBUG:boto:Params: {} 2024-03-04T16:33:37.987 DEBUG:boto:Token: None 2024-03-04T16:33:37.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2749d02c211b6b3-0065e5f7e1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:37.989 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:37.989 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:37.989 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.989 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.989 DEBUG:boto:Method: GET 2024-03-04T16:33:37.989 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.989 DEBUG:boto:Data: 2024-03-04T16:33:37.990 DEBUG:boto:Headers: {} 2024-03-04T16:33:37.990 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:37.990 DEBUG:boto:Port: 8000 2024-03-04T16:33:37.990 DEBUG:boto:Params: {} 2024-03-04T16:33:37.990 DEBUG:boto:Token: None 2024-03-04T16:33:37.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:37 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:37.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI= 2024-03-04T16:33:37.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DpDu9OVL7Hko3/l6SBizohU1pPI=', 'Content-Length': '0'} 2024-03-04T16:33:37.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8d8fde458a7ca49-0065e5f7e1-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.032 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.032 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.032 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.033 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.033 DEBUG:boto:Method: GET 2024-03-04T16:33:38.033 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.033 DEBUG:boto:Data: 2024-03-04T16:33:38.033 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.033 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.033 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.033 DEBUG:boto:Params: {} 2024-03-04T16:33:38.033 DEBUG:boto:Token: None 2024-03-04T16:33:38.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e066709f4944756-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.036 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.037 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.037 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.037 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.037 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.037 DEBUG:boto:Method: GET 2024-03-04T16:33:38.037 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.037 DEBUG:boto:Data: 2024-03-04T16:33:38.037 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.038 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.038 DEBUG:boto:Params: {} 2024-03-04T16:33:38.038 DEBUG:boto:Token: None 2024-03-04T16:33:38.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0965bba7c9bf8d-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.041 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.041 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.042 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.042 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.042 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.042 DEBUG:boto:Method: GET 2024-03-04T16:33:38.042 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.042 DEBUG:boto:Data: 2024-03-04T16:33:38.042 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.042 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.042 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.042 DEBUG:boto:Params: {} 2024-03-04T16:33:38.042 DEBUG:boto:Token: None 2024-03-04T16:33:38.043 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.043 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE= 2024-03-04T16:33:38.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE=', 'Content-Length': '0'} 2024-03-04T16:33:38.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb1805cddfece4a5-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.088 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.088 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.088 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.089 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.089 DEBUG:boto:Method: GET 2024-03-04T16:33:38.089 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.089 DEBUG:boto:Data: 2024-03-04T16:33:38.089 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.089 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.089 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.089 DEBUG:boto:Params: {} 2024-03-04T16:33:38.089 DEBUG:boto:Token: None 2024-03-04T16:33:38.089 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE= 2024-03-04T16:33:38.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE=', 'Content-Length': '0'} 2024-03-04T16:33:38.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a417b5eb910ba0c-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.092 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.092 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.092 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.092 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.093 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.093 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.093 DEBUG:boto:Data: 2024-03-04T16:33:38.093 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.093 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.093 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.093 DEBUG:boto:Params: {} 2024-03-04T16:33:38.093 DEBUG:boto:Token: None 2024-03-04T16:33:38.093 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.093 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jbGtdYLGs7hJ+QQYkAfnhOP1xTw= 2024-03-04T16:33:38.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jbGtdYLGs7hJ+QQYkAfnhOP1xTw=', 'Content-Length': '0'} 2024-03-04T16:33:38.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a18ac0fbb2df2e16-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.095 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.095 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.095 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.096 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.096 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.096 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.096 DEBUG:boto:Data: 2024-03-04T16:33:38.096 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.096 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.096 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.096 DEBUG:boto:Params: {} 2024-03-04T16:33:38.096 DEBUG:boto:Token: None 2024-03-04T16:33:38.096 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.097 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jbGtdYLGs7hJ+QQYkAfnhOP1xTw= 2024-03-04T16:33:38.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jbGtdYLGs7hJ+QQYkAfnhOP1xTw=', 'Content-Length': '0'} 2024-03-04T16:33:38.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6b7074d79a78d82-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.099 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.099 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.099 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.099 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.099 DEBUG:boto:Method: GET 2024-03-04T16:33:38.099 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.099 DEBUG:boto:Data: 2024-03-04T16:33:38.099 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.099 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.099 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.099 DEBUG:boto:Params: {} 2024-03-04T16:33:38.100 DEBUG:boto:Token: None 2024-03-04T16:33:38.100 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.100 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE= 2024-03-04T16:33:38.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE=', 'Content-Length': '0'} 2024-03-04T16:33:38.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050900fbc01b31993-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.102 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.102 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.102 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.102 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.102 DEBUG:boto:Method: GET 2024-03-04T16:33:38.102 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.102 DEBUG:boto:Data: 2024-03-04T16:33:38.102 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.103 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.103 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.103 DEBUG:boto:Params: {} 2024-03-04T16:33:38.103 DEBUG:boto:Token: None 2024-03-04T16:33:38.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:38.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE= 2024-03-04T16:33:38.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kbZFAIle4GMomPYIQWm8kZvI/cE=', 'Content-Length': '0'} 2024-03-04T16:33:38.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ad683e4278181bc-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.105 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.105 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.105 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.105 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.105 DEBUG:boto:Method: GET 2024-03-04T16:33:38.105 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.105 DEBUG:boto:Data: 2024-03-04T16:33:38.105 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.106 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.106 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.106 DEBUG:boto:Params: {} 2024-03-04T16:33:38.106 DEBUG:boto:Token: None 2024-03-04T16:33:38.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.106 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa483ce95010eadb-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.108 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.109 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.109 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.109 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.109 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.109 DEBUG:boto:Method: GET 2024-03-04T16:33:38.109 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.109 DEBUG:boto:Data: 2024-03-04T16:33:38.110 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.110 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.110 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.110 DEBUG:boto:Params: {} 2024-03-04T16:33:38.110 DEBUG:boto:Token: None 2024-03-04T16:33:38.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030032dc38156f98e-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.113 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.113 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.113 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.113 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.113 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.113 DEBUG:boto:Data: 2024-03-04T16:33:38.114 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.114 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.114 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.114 DEBUG:boto:Params: {} 2024-03-04T16:33:38.114 DEBUG:boto:Token: None 2024-03-04T16:33:38.114 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.114 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009060b6c7397f0a28-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.117 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.118 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.118 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.118 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.118 DEBUG:boto:Data: 2024-03-04T16:33:38.118 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.118 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.118 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.118 DEBUG:boto:Params: {} 2024-03-04T16:33:38.118 DEBUG:boto:Token: None 2024-03-04T16:33:38.118 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.119 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5b285381fcb1fc-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.122 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.122 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.122 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.122 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.122 DEBUG:boto:Method: GET 2024-03-04T16:33:38.122 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.123 DEBUG:boto:Data: 2024-03-04T16:33:38.123 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.123 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.123 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.123 DEBUG:boto:Params: {} 2024-03-04T16:33:38.123 DEBUG:boto:Token: None 2024-03-04T16:33:38.123 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.123 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4e7ff3a62ae31b2-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.168 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.172 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.172 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.172 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.172 DEBUG:boto:Method: GET 2024-03-04T16:33:38.172 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.172 DEBUG:boto:Data: 2024-03-04T16:33:38.172 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.172 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.172 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.173 DEBUG:boto:Params: {} 2024-03-04T16:33:38.173 DEBUG:boto:Token: None 2024-03-04T16:33:38.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298922beae6c3477-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.175 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.175 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.175 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.175 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.176 DEBUG:boto:Method: GET 2024-03-04T16:33:38.176 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.176 DEBUG:boto:Data: 2024-03-04T16:33:38.176 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.176 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.176 DEBUG:boto:Params: {} 2024-03-04T16:33:38.176 DEBUG:boto:Token: None 2024-03-04T16:33:38.176 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.176 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf639ec26d0cd0f0-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.179 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.180 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.180 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.180 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.180 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.180 DEBUG:boto:Method: GET 2024-03-04T16:33:38.180 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.180 DEBUG:boto:Data: 2024-03-04T16:33:38.180 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.180 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.181 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.181 DEBUG:boto:Params: {} 2024-03-04T16:33:38.181 DEBUG:boto:Token: None 2024-03-04T16:33:38.181 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.181 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f708df00f2d360c-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.184 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.184 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.184 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.185 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.185 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.185 DEBUG:boto:Method: GET 2024-03-04T16:33:38.185 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.185 DEBUG:boto:Data: 2024-03-04T16:33:38.185 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.185 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.185 DEBUG:boto:Params: {} 2024-03-04T16:33:38.185 DEBUG:boto:Token: None 2024-03-04T16:33:38.185 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY= 2024-03-04T16:33:38.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY=', 'Content-Length': '0'} 2024-03-04T16:33:38.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067ec7a920d15d70b-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.188 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.188 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.188 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.188 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.188 DEBUG:boto:Method: GET 2024-03-04T16:33:38.188 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.188 DEBUG:boto:Data: 2024-03-04T16:33:38.188 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.188 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.188 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.189 DEBUG:boto:Params: {} 2024-03-04T16:33:38.189 DEBUG:boto:Token: None 2024-03-04T16:33:38.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY= 2024-03-04T16:33:38.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY=', 'Content-Length': '0'} 2024-03-04T16:33:38.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056bb4ba9e9dd3cb3-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.192 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.192 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.192 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.192 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.192 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.192 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.192 DEBUG:boto:Data: 2024-03-04T16:33:38.192 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.192 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.192 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.193 DEBUG:boto:Params: {} 2024-03-04T16:33:38.193 DEBUG:boto:Token: None 2024-03-04T16:33:38.193 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m0ip25u/uSDGNusZtcwExlX6jgM= 2024-03-04T16:33:38.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m0ip25u/uSDGNusZtcwExlX6jgM=', 'Content-Length': '0'} 2024-03-04T16:33:38.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000502d0f043bb6bbfd-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.196 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.196 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.196 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.196 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.196 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.196 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.196 DEBUG:boto:Data: 2024-03-04T16:33:38.197 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.197 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.197 DEBUG:boto:Params: {} 2024-03-04T16:33:38.197 DEBUG:boto:Token: None 2024-03-04T16:33:38.197 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.197 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m0ip25u/uSDGNusZtcwExlX6jgM= 2024-03-04T16:33:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m0ip25u/uSDGNusZtcwExlX6jgM=', 'Content-Length': '0'} 2024-03-04T16:33:38.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a391ff98fe0a9a3-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.200 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.200 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.200 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.200 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.200 DEBUG:boto:Method: GET 2024-03-04T16:33:38.200 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.200 DEBUG:boto:Data: 2024-03-04T16:33:38.200 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.200 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.200 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.200 DEBUG:boto:Params: {} 2024-03-04T16:33:38.201 DEBUG:boto:Token: None 2024-03-04T16:33:38.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.201 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY= 2024-03-04T16:33:38.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY=', 'Content-Length': '0'} 2024-03-04T16:33:38.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b3dc961bdeb22fc-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.203 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.203 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.203 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.203 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.204 DEBUG:boto:Method: GET 2024-03-04T16:33:38.204 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.204 DEBUG:boto:Data: 2024-03-04T16:33:38.204 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.204 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.204 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.205 DEBUG:boto:Params: {} 2024-03-04T16:33:38.205 DEBUG:boto:Token: None 2024-03-04T16:33:38.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:38.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY= 2024-03-04T16:33:38.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aYGpVQ6hMidvZFmpLEH5b/TmXdY=', 'Content-Length': '0'} 2024-03-04T16:33:38.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008be007d892c86919-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.207 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.207 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.207 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.207 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.207 DEBUG:boto:Method: GET 2024-03-04T16:33:38.208 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.208 DEBUG:boto:Data: 2024-03-04T16:33:38.208 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.208 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.208 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.208 DEBUG:boto:Params: {} 2024-03-04T16:33:38.208 DEBUG:boto:Token: None 2024-03-04T16:33:38.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.208 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58c60dfd0296829-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.211 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.211 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.211 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.212 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.212 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.212 DEBUG:boto:Method: GET 2024-03-04T16:33:38.212 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.212 DEBUG:boto:Data: 2024-03-04T16:33:38.212 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.212 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.212 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.212 DEBUG:boto:Params: {} 2024-03-04T16:33:38.212 DEBUG:boto:Token: None 2024-03-04T16:33:38.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.213 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a79be6753a2185a-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.215 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.216 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.216 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.216 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.216 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.216 DEBUG:boto:Data: 2024-03-04T16:33:38.216 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.216 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.217 DEBUG:boto:Params: {} 2024-03-04T16:33:38.217 DEBUG:boto:Token: None 2024-03-04T16:33:38.217 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.217 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a535a80436a547c-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.220 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.220 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.221 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.221 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.221 DEBUG:boto:Data: 2024-03-04T16:33:38.221 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.221 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.221 DEBUG:boto:Params: {} 2024-03-04T16:33:38.221 DEBUG:boto:Token: None 2024-03-04T16:33:38.221 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.221 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ab7d8eaad270d1d-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.224 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.225 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.225 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.225 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.225 DEBUG:boto:Method: GET 2024-03-04T16:33:38.225 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.225 DEBUG:boto:Data: 2024-03-04T16:33:38.225 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.225 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.225 DEBUG:boto:Params: {} 2024-03-04T16:33:38.225 DEBUG:boto:Token: None 2024-03-04T16:33:38.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5e7fcd32b72cbf3-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.228 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.228 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.228 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.228 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.228 DEBUG:boto:Method: GET 2024-03-04T16:33:38.228 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.228 DEBUG:boto:Data: 2024-03-04T16:33:38.228 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.228 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.228 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.229 DEBUG:boto:Params: {} 2024-03-04T16:33:38.229 DEBUG:boto:Token: None 2024-03-04T16:33:38.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060b1170c11fd78e4-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.231 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.231 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.232 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.232 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.232 DEBUG:boto:Method: GET 2024-03-04T16:33:38.232 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.232 DEBUG:boto:Data: 2024-03-04T16:33:38.232 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.232 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.232 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.232 DEBUG:boto:Params: {} 2024-03-04T16:33:38.232 DEBUG:boto:Token: None 2024-03-04T16:33:38.233 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008435fe27d8ac0112-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.235 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.236 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.236 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.236 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.236 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.236 DEBUG:boto:Method: GET 2024-03-04T16:33:38.236 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.236 DEBUG:boto:Data: 2024-03-04T16:33:38.236 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.236 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.236 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.237 DEBUG:boto:Params: {} 2024-03-04T16:33:38.237 DEBUG:boto:Token: None 2024-03-04T16:33:38.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5bfaa557d93b46b-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.240 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.240 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.240 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.240 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.240 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.241 DEBUG:boto:Method: GET 2024-03-04T16:33:38.241 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.241 DEBUG:boto:Data: 2024-03-04T16:33:38.241 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.241 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.241 DEBUG:boto:Params: {} 2024-03-04T16:33:38.241 DEBUG:boto:Token: None 2024-03-04T16:33:38.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0= 2024-03-04T16:33:38.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0=', 'Content-Length': '0'} 2024-03-04T16:33:38.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d38b17cad3652b28-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.244 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.244 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.244 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.244 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.244 DEBUG:boto:Method: GET 2024-03-04T16:33:38.244 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.244 DEBUG:boto:Data: 2024-03-04T16:33:38.244 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.245 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.245 DEBUG:boto:Params: {} 2024-03-04T16:33:38.245 DEBUG:boto:Token: None 2024-03-04T16:33:38.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0= 2024-03-04T16:33:38.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0=', 'Content-Length': '0'} 2024-03-04T16:33:38.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0654c82d372ef96-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.247 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.247 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.248 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.248 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.248 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.248 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.248 DEBUG:boto:Data: 2024-03-04T16:33:38.248 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.248 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.248 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.249 DEBUG:boto:Params: {} 2024-03-04T16:33:38.249 DEBUG:boto:Token: None 2024-03-04T16:33:38.249 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5D94Wpy9e6IrvF/RFVp+hXyw1X0= 2024-03-04T16:33:38.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5D94Wpy9e6IrvF/RFVp+hXyw1X0=', 'Content-Length': '0'} 2024-03-04T16:33:38.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d72f778b7e08ef6-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.251 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.251 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.251 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.251 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.251 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.251 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.251 DEBUG:boto:Data: 2024-03-04T16:33:38.252 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.252 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.252 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.252 DEBUG:boto:Params: {} 2024-03-04T16:33:38.252 DEBUG:boto:Token: None 2024-03-04T16:33:38.252 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.252 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5D94Wpy9e6IrvF/RFVp+hXyw1X0= 2024-03-04T16:33:38.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5D94Wpy9e6IrvF/RFVp+hXyw1X0=', 'Content-Length': '0'} 2024-03-04T16:33:38.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000338281c52c220a13-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.254 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.254 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.254 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.255 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.255 DEBUG:boto:Method: GET 2024-03-04T16:33:38.255 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.255 DEBUG:boto:Data: 2024-03-04T16:33:38.255 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.255 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.255 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.255 DEBUG:boto:Params: {} 2024-03-04T16:33:38.255 DEBUG:boto:Token: None 2024-03-04T16:33:38.255 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.256 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0= 2024-03-04T16:33:38.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0=', 'Content-Length': '0'} 2024-03-04T16:33:38.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec2523b74406bc41-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.258 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.258 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.258 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.258 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.258 DEBUG:boto:Method: GET 2024-03-04T16:33:38.258 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.258 DEBUG:boto:Data: 2024-03-04T16:33:38.258 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.258 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.258 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.259 DEBUG:boto:Params: {} 2024-03-04T16:33:38.259 DEBUG:boto:Token: None 2024-03-04T16:33:38.259 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:38.259 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0= 2024-03-04T16:33:38.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:66WUj4cb7bI0kFjM8JBEFHbLD/0=', 'Content-Length': '0'} 2024-03-04T16:33:38.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080dd2edeeeedb789-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.261 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.261 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.261 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.261 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.261 DEBUG:boto:Method: GET 2024-03-04T16:33:38.261 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.261 DEBUG:boto:Data: 2024-03-04T16:33:38.261 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.262 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.262 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.262 DEBUG:boto:Params: {} 2024-03-04T16:33:38.262 DEBUG:boto:Token: None 2024-03-04T16:33:38.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.262 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332db2432d6e8828-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.264 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.265 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.265 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.265 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.265 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.265 DEBUG:boto:Method: GET 2024-03-04T16:33:38.265 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.265 DEBUG:boto:Data: 2024-03-04T16:33:38.265 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.265 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.266 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.266 DEBUG:boto:Params: {} 2024-03-04T16:33:38.266 DEBUG:boto:Token: None 2024-03-04T16:33:38.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d69926e067f4cb-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.269 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.269 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.269 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.269 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.269 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.269 DEBUG:boto:Data: 2024-03-04T16:33:38.270 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.270 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.270 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.270 DEBUG:boto:Params: {} 2024-03-04T16:33:38.270 DEBUG:boto:Token: None 2024-03-04T16:33:38.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b0501afbf509ec6-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.273 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.273 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.273 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.273 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.274 DEBUG:boto:Data: 2024-03-04T16:33:38.274 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.274 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.274 DEBUG:boto:Params: {} 2024-03-04T16:33:38.274 DEBUG:boto:Token: None 2024-03-04T16:33:38.274 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005df6151715ee8645-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.277 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.277 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.278 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.278 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.278 DEBUG:boto:Method: GET 2024-03-04T16:33:38.278 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.278 DEBUG:boto:Data: 2024-03-04T16:33:38.278 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.278 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.278 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.278 DEBUG:boto:Params: {} 2024-03-04T16:33:38.278 DEBUG:boto:Token: None 2024-03-04T16:33:38.279 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.279 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004214efb4d0b7f20d-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.324 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.324 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.324 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.324 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.325 DEBUG:boto:Method: GET 2024-03-04T16:33:38.325 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.325 DEBUG:boto:Data: 2024-03-04T16:33:38.325 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.325 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.325 DEBUG:boto:Params: {} 2024-03-04T16:33:38.325 DEBUG:boto:Token: None 2024-03-04T16:33:38.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f0243431f156aa8-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.328 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.328 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.328 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.328 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.328 DEBUG:boto:Method: GET 2024-03-04T16:33:38.328 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.328 DEBUG:boto:Data: 2024-03-04T16:33:38.328 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.329 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.329 DEBUG:boto:Params: {} 2024-03-04T16:33:38.329 DEBUG:boto:Token: None 2024-03-04T16:33:38.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c8a86b3230ba00-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.332 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.332 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.332 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.333 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.333 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.333 DEBUG:boto:Method: GET 2024-03-04T16:33:38.333 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.333 DEBUG:boto:Data: 2024-03-04T16:33:38.333 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.333 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.333 DEBUG:boto:Params: {} 2024-03-04T16:33:38.333 DEBUG:boto:Token: None 2024-03-04T16:33:38.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.334 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000294153b2dab736b5-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.336 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.337 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.337 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.337 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.337 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.337 DEBUG:boto:Method: GET 2024-03-04T16:33:38.337 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.337 DEBUG:boto:Data: 2024-03-04T16:33:38.337 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.338 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.338 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.338 DEBUG:boto:Params: {} 2024-03-04T16:33:38.338 DEBUG:boto:Token: None 2024-03-04T16:33:38.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo= 2024-03-04T16:33:38.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo=', 'Content-Length': '0'} 2024-03-04T16:33:38.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000461f9488ee245920-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.340 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.340 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.340 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.341 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.341 DEBUG:boto:Method: GET 2024-03-04T16:33:38.341 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.341 DEBUG:boto:Data: 2024-03-04T16:33:38.341 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.341 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.341 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.341 DEBUG:boto:Params: {} 2024-03-04T16:33:38.341 DEBUG:boto:Token: None 2024-03-04T16:33:38.341 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.341 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo= 2024-03-04T16:33:38.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo=', 'Content-Length': '0'} 2024-03-04T16:33:38.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026ed46752ba4e80b-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.344 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.344 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.344 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.344 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.344 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.344 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.344 DEBUG:boto:Data: 2024-03-04T16:33:38.344 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.344 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.344 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.344 DEBUG:boto:Params: {} 2024-03-04T16:33:38.345 DEBUG:boto:Token: None 2024-03-04T16:33:38.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ug1RIyhwp5t7fP9aMWcfvh0BQ+A= 2024-03-04T16:33:38.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ug1RIyhwp5t7fP9aMWcfvh0BQ+A=', 'Content-Length': '0'} 2024-03-04T16:33:38.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000062064ad58e1e8f0-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.347 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.347 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.347 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.347 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.347 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.347 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.347 DEBUG:boto:Data: 2024-03-04T16:33:38.347 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.347 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.347 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.347 DEBUG:boto:Params: {} 2024-03-04T16:33:38.348 DEBUG:boto:Token: None 2024-03-04T16:33:38.348 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.348 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ug1RIyhwp5t7fP9aMWcfvh0BQ+A= 2024-03-04T16:33:38.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ug1RIyhwp5t7fP9aMWcfvh0BQ+A=', 'Content-Length': '0'} 2024-03-04T16:33:38.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec1b3067a9a4b8bc-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.350 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.350 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.350 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.350 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.350 DEBUG:boto:Method: GET 2024-03-04T16:33:38.351 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.351 DEBUG:boto:Data: 2024-03-04T16:33:38.351 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.351 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.351 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.351 DEBUG:boto:Params: {} 2024-03-04T16:33:38.351 DEBUG:boto:Token: None 2024-03-04T16:33:38.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.351 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo= 2024-03-04T16:33:38.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo=', 'Content-Length': '0'} 2024-03-04T16:33:38.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032ec1dd00b82c5a7-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.400 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.400 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.400 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.401 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.401 DEBUG:boto:Method: GET 2024-03-04T16:33:38.401 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.401 DEBUG:boto:Data: 2024-03-04T16:33:38.401 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.401 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.401 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.401 DEBUG:boto:Params: {} 2024-03-04T16:33:38.401 DEBUG:boto:Token: None 2024-03-04T16:33:38.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:38.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo= 2024-03-04T16:33:38.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mhnyKoGhUFJvU/jYsJkKVywI8yo=', 'Content-Length': '0'} 2024-03-04T16:33:38.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9a33b6dd09b9577-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.404 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.404 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.404 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.404 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.404 DEBUG:boto:Method: GET 2024-03-04T16:33:38.404 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.405 DEBUG:boto:Data: 2024-03-04T16:33:38.405 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.405 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.405 DEBUG:boto:Params: {} 2024-03-04T16:33:38.405 DEBUG:boto:Token: None 2024-03-04T16:33:38.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a134dfe2a7e85ef-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.407 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.408 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.408 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.408 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.408 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.408 DEBUG:boto:Method: GET 2024-03-04T16:33:38.408 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.408 DEBUG:boto:Data: 2024-03-04T16:33:38.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.409 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.409 DEBUG:boto:Params: {} 2024-03-04T16:33:38.409 DEBUG:boto:Token: None 2024-03-04T16:33:38.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2c9807e0b0bbf2-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.412 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.412 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.412 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.412 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.412 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.413 DEBUG:boto:Data: 2024-03-04T16:33:38.413 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.413 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.413 DEBUG:boto:Params: {} 2024-03-04T16:33:38.413 DEBUG:boto:Token: None 2024-03-04T16:33:38.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006000127bc97c34e9-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.416 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.417 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.417 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.417 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.417 DEBUG:boto:Data: 2024-03-04T16:33:38.417 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.417 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.417 DEBUG:boto:Params: {} 2024-03-04T16:33:38.417 DEBUG:boto:Token: None 2024-03-04T16:33:38.417 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000339d4b3068f59b3-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.421 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.421 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.421 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.421 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.421 DEBUG:boto:Method: GET 2024-03-04T16:33:38.421 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.421 DEBUG:boto:Data: 2024-03-04T16:33:38.421 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.421 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.421 DEBUG:boto:Params: {} 2024-03-04T16:33:38.422 DEBUG:boto:Token: None 2024-03-04T16:33:38.422 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059afb945589b812e-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.424 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.424 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.424 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.424 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.424 DEBUG:boto:Method: GET 2024-03-04T16:33:38.424 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.424 DEBUG:boto:Data: 2024-03-04T16:33:38.424 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.424 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.424 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.425 DEBUG:boto:Params: {} 2024-03-04T16:33:38.425 DEBUG:boto:Token: None 2024-03-04T16:33:38.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000246266815ee64ce7-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.427 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.427 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.427 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.427 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.428 DEBUG:boto:Method: GET 2024-03-04T16:33:38.428 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.428 DEBUG:boto:Data: 2024-03-04T16:33:38.428 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.428 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.428 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.428 DEBUG:boto:Params: {} 2024-03-04T16:33:38.428 DEBUG:boto:Token: None 2024-03-04T16:33:38.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.428 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8ac4fe36ec4294-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.431 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.431 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.431 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.431 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.432 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.432 DEBUG:boto:Method: GET 2024-03-04T16:33:38.432 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.432 DEBUG:boto:Data: 2024-03-04T16:33:38.432 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.432 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.432 DEBUG:boto:Params: {} 2024-03-04T16:33:38.432 DEBUG:boto:Token: None 2024-03-04T16:33:38.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de32cc3b3934f01b-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.435 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.436 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.436 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.436 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.436 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.436 DEBUG:boto:Method: GET 2024-03-04T16:33:38.436 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.436 DEBUG:boto:Data: 2024-03-04T16:33:38.436 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.436 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.436 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.436 DEBUG:boto:Params: {} 2024-03-04T16:33:38.437 DEBUG:boto:Token: None 2024-03-04T16:33:38.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM= 2024-03-04T16:33:38.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM=', 'Content-Length': '0'} 2024-03-04T16:33:38.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090a42c8bd242e460-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.439 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.439 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.439 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.439 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.439 DEBUG:boto:Method: GET 2024-03-04T16:33:38.439 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.439 DEBUG:boto:Data: 2024-03-04T16:33:38.440 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.440 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.440 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.440 DEBUG:boto:Params: {} 2024-03-04T16:33:38.440 DEBUG:boto:Token: None 2024-03-04T16:33:38.440 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.440 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM= 2024-03-04T16:33:38.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM=', 'Content-Length': '0'} 2024-03-04T16:33:38.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a89948a52c99578-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.442 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.442 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.442 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.442 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.443 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.443 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.443 DEBUG:boto:Data: 2024-03-04T16:33:38.443 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.443 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.443 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.443 DEBUG:boto:Params: {} 2024-03-04T16:33:38.443 DEBUG:boto:Token: None 2024-03-04T16:33:38.443 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rmrArskxmg0ppD77ryZ3ovdtcg0= 2024-03-04T16:33:38.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rmrArskxmg0ppD77ryZ3ovdtcg0=', 'Content-Length': '0'} 2024-03-04T16:33:38.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f977ca7e694c516-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.445 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.445 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.445 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.446 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.446 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.446 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.446 DEBUG:boto:Data: 2024-03-04T16:33:38.446 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.446 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.446 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.446 DEBUG:boto:Params: {} 2024-03-04T16:33:38.446 DEBUG:boto:Token: None 2024-03-04T16:33:38.446 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rmrArskxmg0ppD77ryZ3ovdtcg0= 2024-03-04T16:33:38.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rmrArskxmg0ppD77ryZ3ovdtcg0=', 'Content-Length': '0'} 2024-03-04T16:33:38.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0cab3dcf4a5297a-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.448 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.449 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.449 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.449 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.449 DEBUG:boto:Method: GET 2024-03-04T16:33:38.449 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.449 DEBUG:boto:Data: 2024-03-04T16:33:38.449 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.449 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.449 DEBUG:boto:Params: {} 2024-03-04T16:33:38.449 DEBUG:boto:Token: None 2024-03-04T16:33:38.450 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM= 2024-03-04T16:33:38.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM=', 'Content-Length': '0'} 2024-03-04T16:33:38.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1f082e4c2846d07-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.452 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.452 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.452 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.452 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.452 DEBUG:boto:Method: GET 2024-03-04T16:33:38.452 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.452 DEBUG:boto:Data: 2024-03-04T16:33:38.452 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.452 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.452 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.452 DEBUG:boto:Params: {} 2024-03-04T16:33:38.453 DEBUG:boto:Token: None 2024-03-04T16:33:38.453 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:38.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM= 2024-03-04T16:33:38.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5owzrPFBZrwjFjb7muYHvs+ZUbM=', 'Content-Length': '0'} 2024-03-04T16:33:38.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ac818fae2a06925-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.500 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.500 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.500 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.500 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.501 DEBUG:boto:Method: GET 2024-03-04T16:33:38.501 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.501 DEBUG:boto:Data: 2024-03-04T16:33:38.501 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.501 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.501 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.501 DEBUG:boto:Params: {} 2024-03-04T16:33:38.501 DEBUG:boto:Token: None 2024-03-04T16:33:38.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebac9d6e7478d151-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.504 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.505 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.505 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.505 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.505 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.505 DEBUG:boto:Method: GET 2024-03-04T16:33:38.505 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.505 DEBUG:boto:Data: 2024-03-04T16:33:38.506 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.506 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.506 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.506 DEBUG:boto:Params: {} 2024-03-04T16:33:38.506 DEBUG:boto:Token: None 2024-03-04T16:33:38.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e5f342d7f38be74-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.509 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.509 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.509 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.509 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.509 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.509 DEBUG:boto:Data: 2024-03-04T16:33:38.510 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.510 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.510 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.510 DEBUG:boto:Params: {} 2024-03-04T16:33:38.510 DEBUG:boto:Token: None 2024-03-04T16:33:38.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000175957bcf7b96d3b-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.513 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.513 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.513 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.514 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.514 DEBUG:boto:Data: 2024-03-04T16:33:38.514 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.514 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.514 DEBUG:boto:Params: {} 2024-03-04T16:33:38.514 DEBUG:boto:Token: None 2024-03-04T16:33:38.514 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042ab530471025622-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.517 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.518 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.518 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.518 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.518 DEBUG:boto:Method: GET 2024-03-04T16:33:38.518 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.518 DEBUG:boto:Data: 2024-03-04T16:33:38.518 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.518 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.518 DEBUG:boto:Params: {} 2024-03-04T16:33:38.518 DEBUG:boto:Token: None 2024-03-04T16:33:38.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c544615e039a9a01-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.568 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.568 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.568 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.568 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.569 DEBUG:boto:Method: GET 2024-03-04T16:33:38.569 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.569 DEBUG:boto:Data: 2024-03-04T16:33:38.569 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.569 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.569 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.569 DEBUG:boto:Params: {} 2024-03-04T16:33:38.569 DEBUG:boto:Token: None 2024-03-04T16:33:38.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0858235783c1848-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.572 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.572 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.572 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.572 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.572 DEBUG:boto:Method: GET 2024-03-04T16:33:38.572 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.572 DEBUG:boto:Data: 2024-03-04T16:33:38.573 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.573 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.573 DEBUG:boto:Params: {} 2024-03-04T16:33:38.573 DEBUG:boto:Token: None 2024-03-04T16:33:38.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9bbbe9642cee6b5-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.576 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.576 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.576 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.576 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.576 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.576 DEBUG:boto:Method: GET 2024-03-04T16:33:38.577 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.577 DEBUG:boto:Data: 2024-03-04T16:33:38.577 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.577 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.577 DEBUG:boto:Params: {} 2024-03-04T16:33:38.577 DEBUG:boto:Token: None 2024-03-04T16:33:38.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e57996100cf34e24-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.584 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.584 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.585 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.585 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.585 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.585 DEBUG:boto:Method: GET 2024-03-04T16:33:38.585 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.585 DEBUG:boto:Data: 2024-03-04T16:33:38.585 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.585 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.585 DEBUG:boto:Params: {} 2024-03-04T16:33:38.585 DEBUG:boto:Token: None 2024-03-04T16:33:38.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk= 2024-03-04T16:33:38.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk=', 'Content-Length': '0'} 2024-03-04T16:33:38.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c5d81346c268b9a-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.588 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.589 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.589 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.589 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.589 DEBUG:boto:Method: GET 2024-03-04T16:33:38.589 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.589 DEBUG:boto:Data: 2024-03-04T16:33:38.589 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.589 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.589 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.590 DEBUG:boto:Params: {} 2024-03-04T16:33:38.590 DEBUG:boto:Token: None 2024-03-04T16:33:38.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk= 2024-03-04T16:33:38.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk=', 'Content-Length': '0'} 2024-03-04T16:33:38.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027417ae0dacf11f4-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.636 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.636 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.636 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.636 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.637 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.637 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.637 DEBUG:boto:Data: 2024-03-04T16:33:38.637 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.637 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.637 DEBUG:boto:Params: {} 2024-03-04T16:33:38.637 DEBUG:boto:Token: None 2024-03-04T16:33:38.637 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0FPZq/7kfbJkMJwFy0BFpDe3WUY= 2024-03-04T16:33:38.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0FPZq/7kfbJkMJwFy0BFpDe3WUY=', 'Content-Length': '0'} 2024-03-04T16:33:38.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033aec2065dce21b0-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.640 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.640 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.640 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.640 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.640 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.640 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.640 DEBUG:boto:Data: 2024-03-04T16:33:38.640 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.640 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.641 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.641 DEBUG:boto:Params: {} 2024-03-04T16:33:38.641 DEBUG:boto:Token: None 2024-03-04T16:33:38.641 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.641 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0FPZq/7kfbJkMJwFy0BFpDe3WUY= 2024-03-04T16:33:38.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0FPZq/7kfbJkMJwFy0BFpDe3WUY=', 'Content-Length': '0'} 2024-03-04T16:33:38.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da76532432c326a0-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.643 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.643 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.643 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.643 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.644 DEBUG:boto:Method: GET 2024-03-04T16:33:38.644 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.644 DEBUG:boto:Data: 2024-03-04T16:33:38.644 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.644 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.644 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.644 DEBUG:boto:Params: {} 2024-03-04T16:33:38.644 DEBUG:boto:Token: None 2024-03-04T16:33:38.644 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.645 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk= 2024-03-04T16:33:38.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk=', 'Content-Length': '0'} 2024-03-04T16:33:38.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000749947b524021929-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.647 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.647 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.647 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.647 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.647 DEBUG:boto:Method: GET 2024-03-04T16:33:38.647 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.647 DEBUG:boto:Data: 2024-03-04T16:33:38.647 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.647 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.647 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.648 DEBUG:boto:Params: {} 2024-03-04T16:33:38.648 DEBUG:boto:Token: None 2024-03-04T16:33:38.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:38.648 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk= 2024-03-04T16:33:38.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1RMxz1tOjhP+P0FBpUfEfzvbTk=', 'Content-Length': '0'} 2024-03-04T16:33:38.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089ebf5d19d019c12-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.650 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.650 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.650 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.650 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.650 DEBUG:boto:Method: GET 2024-03-04T16:33:38.651 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.651 DEBUG:boto:Data: 2024-03-04T16:33:38.651 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.651 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.651 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.651 DEBUG:boto:Params: {} 2024-03-04T16:33:38.651 DEBUG:boto:Token: None 2024-03-04T16:33:38.651 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.651 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd75cad78adbf9f7-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.654 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.654 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.654 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.654 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.655 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.655 DEBUG:boto:Method: GET 2024-03-04T16:33:38.655 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.655 DEBUG:boto:Data: 2024-03-04T16:33:38.655 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.655 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.655 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.655 DEBUG:boto:Params: {} 2024-03-04T16:33:38.655 DEBUG:boto:Token: None 2024-03-04T16:33:38.655 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.656 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a3fcc5dfc06f2cf-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.658 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.658 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.658 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.659 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.659 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.659 DEBUG:boto:Data: 2024-03-04T16:33:38.659 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.659 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.659 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.660 DEBUG:boto:Params: {} 2024-03-04T16:33:38.660 DEBUG:boto:Token: None 2024-03-04T16:33:38.660 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0a1b120ec642af8-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.663 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.663 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.663 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.663 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.664 DEBUG:boto:Data: 2024-03-04T16:33:38.664 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.664 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.664 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.664 DEBUG:boto:Params: {} 2024-03-04T16:33:38.664 DEBUG:boto:Token: None 2024-03-04T16:33:38.664 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.664 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000717502feac5cd5af-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.668 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.668 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.668 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.668 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.668 DEBUG:boto:Method: GET 2024-03-04T16:33:38.668 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.668 DEBUG:boto:Data: 2024-03-04T16:33:38.668 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.669 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.669 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.669 DEBUG:boto:Params: {} 2024-03-04T16:33:38.669 DEBUG:boto:Token: None 2024-03-04T16:33:38.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c3fcedf714fe262-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.671 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.671 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.672 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.672 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.672 DEBUG:boto:Method: GET 2024-03-04T16:33:38.672 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.672 DEBUG:boto:Data: 2024-03-04T16:33:38.672 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.672 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.672 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.672 DEBUG:boto:Params: {} 2024-03-04T16:33:38.672 DEBUG:boto:Token: None 2024-03-04T16:33:38.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.673 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d59993dd55165ad-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.675 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.675 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.675 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.675 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.675 DEBUG:boto:Method: GET 2024-03-04T16:33:38.675 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.675 DEBUG:boto:Data: 2024-03-04T16:33:38.676 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.676 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.676 DEBUG:boto:Params: {} 2024-03-04T16:33:38.676 DEBUG:boto:Token: None 2024-03-04T16:33:38.676 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.676 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1dc3a659bc8909-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.679 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.679 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.679 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.679 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.679 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.679 DEBUG:boto:Method: GET 2024-03-04T16:33:38.679 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.680 DEBUG:boto:Data: 2024-03-04T16:33:38.680 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.680 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.680 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.680 DEBUG:boto:Params: {} 2024-03-04T16:33:38.680 DEBUG:boto:Token: None 2024-03-04T16:33:38.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.680 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf1e44a05be2a51-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.683 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.683 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.683 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.683 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.683 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.683 DEBUG:boto:Method: GET 2024-03-04T16:33:38.684 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.684 DEBUG:boto:Data: 2024-03-04T16:33:38.684 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.684 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.684 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.684 DEBUG:boto:Params: {} 2024-03-04T16:33:38.684 DEBUG:boto:Token: None 2024-03-04T16:33:38.684 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.684 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM= 2024-03-04T16:33:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM=', 'Content-Length': '0'} 2024-03-04T16:33:38.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f473ffa605705d3-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.686 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.687 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.687 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.687 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.687 DEBUG:boto:Method: GET 2024-03-04T16:33:38.687 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.687 DEBUG:boto:Data: 2024-03-04T16:33:38.687 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.687 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.687 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.687 DEBUG:boto:Params: {} 2024-03-04T16:33:38.688 DEBUG:boto:Token: None 2024-03-04T16:33:38.688 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.688 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM= 2024-03-04T16:33:38.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM=', 'Content-Length': '0'} 2024-03-04T16:33:38.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faf45e3e16e10d1e-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.690 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.690 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.690 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.690 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.690 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.690 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.690 DEBUG:boto:Data: 2024-03-04T16:33:38.690 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.691 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.691 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.691 DEBUG:boto:Params: {} 2024-03-04T16:33:38.691 DEBUG:boto:Token: None 2024-03-04T16:33:38.691 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.691 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:v24FOXtU0NhiXyAyrFSqkM7qRCo= 2024-03-04T16:33:38.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v24FOXtU0NhiXyAyrFSqkM7qRCo=', 'Content-Length': '0'} 2024-03-04T16:33:38.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a33803d0d8278060-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.693 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.693 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.693 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.693 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.693 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.693 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.694 DEBUG:boto:Data: 2024-03-04T16:33:38.694 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.694 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.694 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.694 DEBUG:boto:Params: {} 2024-03-04T16:33:38.694 DEBUG:boto:Token: None 2024-03-04T16:33:38.694 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:v24FOXtU0NhiXyAyrFSqkM7qRCo= 2024-03-04T16:33:38.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v24FOXtU0NhiXyAyrFSqkM7qRCo=', 'Content-Length': '0'} 2024-03-04T16:33:38.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000839311e6caae72a1-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.696 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.696 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.697 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.697 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.697 DEBUG:boto:Method: GET 2024-03-04T16:33:38.697 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.697 DEBUG:boto:Data: 2024-03-04T16:33:38.697 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.697 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.697 DEBUG:boto:Params: {} 2024-03-04T16:33:38.697 DEBUG:boto:Token: None 2024-03-04T16:33:38.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM= 2024-03-04T16:33:38.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM=', 'Content-Length': '0'} 2024-03-04T16:33:38.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8ef3140fb5bde62-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.700 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.700 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.700 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.700 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.700 DEBUG:boto:Method: GET 2024-03-04T16:33:38.700 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.701 DEBUG:boto:Data: 2024-03-04T16:33:38.701 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.701 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.701 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.701 DEBUG:boto:Params: {} 2024-03-04T16:33:38.701 DEBUG:boto:Token: None 2024-03-04T16:33:38.701 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:38.701 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM= 2024-03-04T16:33:38.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kz13TGK30+dp0+HJmEZp/SvgLjM=', 'Content-Length': '0'} 2024-03-04T16:33:38.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000862ffec2593f3f42-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.703 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.704 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.704 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.704 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.704 DEBUG:boto:Method: GET 2024-03-04T16:33:38.704 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.704 DEBUG:boto:Data: 2024-03-04T16:33:38.704 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.704 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.704 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.704 DEBUG:boto:Params: {} 2024-03-04T16:33:38.705 DEBUG:boto:Token: None 2024-03-04T16:33:38.705 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.705 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095098d41780dbdd9-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.708 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.708 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.708 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.708 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.708 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.708 DEBUG:boto:Method: GET 2024-03-04T16:33:38.708 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.709 DEBUG:boto:Data: 2024-03-04T16:33:38.709 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.709 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.709 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.709 DEBUG:boto:Params: {} 2024-03-04T16:33:38.709 DEBUG:boto:Token: None 2024-03-04T16:33:38.709 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.709 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8f71b31ca7d483-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.712 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.712 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.712 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.712 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.713 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.713 DEBUG:boto:Data: 2024-03-04T16:33:38.713 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.713 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.713 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.713 DEBUG:boto:Params: {} 2024-03-04T16:33:38.713 DEBUG:boto:Token: None 2024-03-04T16:33:38.713 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.713 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dfab02bb29e3397-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.717 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.717 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.717 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.717 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.717 DEBUG:boto:Data: 2024-03-04T16:33:38.718 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.718 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.718 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.718 DEBUG:boto:Params: {} 2024-03-04T16:33:38.718 DEBUG:boto:Token: None 2024-03-04T16:33:38.718 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4a7ce84d7332c10-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.721 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.722 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.722 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.722 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.722 DEBUG:boto:Method: GET 2024-03-04T16:33:38.722 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.722 DEBUG:boto:Data: 2024-03-04T16:33:38.722 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.722 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.722 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.722 DEBUG:boto:Params: {} 2024-03-04T16:33:38.723 DEBUG:boto:Token: None 2024-03-04T16:33:38.723 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.723 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f827bbdc10a0ed5-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.772 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.772 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.772 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.772 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.773 DEBUG:boto:Method: GET 2024-03-04T16:33:38.773 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.773 DEBUG:boto:Data: 2024-03-04T16:33:38.773 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.773 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.773 DEBUG:boto:Params: {} 2024-03-04T16:33:38.773 DEBUG:boto:Token: None 2024-03-04T16:33:38.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee48cf38a3199662-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.776 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.776 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.776 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.776 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.776 DEBUG:boto:Method: GET 2024-03-04T16:33:38.776 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.776 DEBUG:boto:Data: 2024-03-04T16:33:38.776 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.777 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.777 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.777 DEBUG:boto:Params: {} 2024-03-04T16:33:38.777 DEBUG:boto:Token: None 2024-03-04T16:33:38.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88208e940288862-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.784 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.785 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.785 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.785 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.785 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.785 DEBUG:boto:Method: GET 2024-03-04T16:33:38.785 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.785 DEBUG:boto:Data: 2024-03-04T16:33:38.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.785 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.785 DEBUG:boto:Params: {} 2024-03-04T16:33:38.786 DEBUG:boto:Token: None 2024-03-04T16:33:38.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ec58251b703a22-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.789 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.790 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.790 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.790 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.790 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.790 DEBUG:boto:Method: GET 2024-03-04T16:33:38.790 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.790 DEBUG:boto:Data: 2024-03-04T16:33:38.790 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.790 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.790 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.791 DEBUG:boto:Params: {} 2024-03-04T16:33:38.791 DEBUG:boto:Token: None 2024-03-04T16:33:38.791 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg= 2024-03-04T16:33:38.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg=', 'Content-Length': '0'} 2024-03-04T16:33:38.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d099ff4a9a2a517e-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.793 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.793 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.793 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.793 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.794 DEBUG:boto:Method: GET 2024-03-04T16:33:38.794 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.794 DEBUG:boto:Data: 2024-03-04T16:33:38.794 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.794 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.794 DEBUG:boto:Params: {} 2024-03-04T16:33:38.794 DEBUG:boto:Token: None 2024-03-04T16:33:38.794 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg= 2024-03-04T16:33:38.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg=', 'Content-Length': '0'} 2024-03-04T16:33:38.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095e70fd111e4eab7-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.844 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.844 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.844 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.845 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.845 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.845 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.845 DEBUG:boto:Data: 2024-03-04T16:33:38.845 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.845 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.845 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.845 DEBUG:boto:Params: {} 2024-03-04T16:33:38.845 DEBUG:boto:Token: None 2024-03-04T16:33:38.846 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pEcyiCDDvDdqcC6Flp3qZUIbFNk= 2024-03-04T16:33:38.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pEcyiCDDvDdqcC6Flp3qZUIbFNk=', 'Content-Length': '0'} 2024-03-04T16:33:38.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000262869c635cf8fab-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.848 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.848 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.848 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.848 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.848 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.849 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.849 DEBUG:boto:Data: 2024-03-04T16:33:38.849 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.849 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.849 DEBUG:boto:Params: {} 2024-03-04T16:33:38.849 DEBUG:boto:Token: None 2024-03-04T16:33:38.849 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pEcyiCDDvDdqcC6Flp3qZUIbFNk= 2024-03-04T16:33:38.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pEcyiCDDvDdqcC6Flp3qZUIbFNk=', 'Content-Length': '0'} 2024-03-04T16:33:38.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7558b2a64f456bd-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.852 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.852 DEBUG:boto:Method: GET 2024-03-04T16:33:38.852 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.852 DEBUG:boto:Data: 2024-03-04T16:33:38.852 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.853 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.853 DEBUG:boto:Params: {} 2024-03-04T16:33:38.853 DEBUG:boto:Token: None 2024-03-04T16:33:38.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg= 2024-03-04T16:33:38.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg=', 'Content-Length': '0'} 2024-03-04T16:33:38.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006340c0eb5371b2fa-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.855 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.855 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.856 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.856 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.856 DEBUG:boto:Method: GET 2024-03-04T16:33:38.856 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.856 DEBUG:boto:Data: 2024-03-04T16:33:38.856 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.856 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.856 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.856 DEBUG:boto:Params: {} 2024-03-04T16:33:38.856 DEBUG:boto:Token: None 2024-03-04T16:33:38.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:38.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg= 2024-03-04T16:33:38.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Jj1K5lRnT7cSSc5fck1exj7vNLg=', 'Content-Length': '0'} 2024-03-04T16:33:38.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008127932595bf095c-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.859 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.859 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.859 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.859 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.859 DEBUG:boto:Method: GET 2024-03-04T16:33:38.859 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.859 DEBUG:boto:Data: 2024-03-04T16:33:38.859 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.860 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.860 DEBUG:boto:Params: {} 2024-03-04T16:33:38.860 DEBUG:boto:Token: None 2024-03-04T16:33:38.860 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9c192fe4043021-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.863 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.864 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.864 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.864 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.864 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.864 DEBUG:boto:Method: GET 2024-03-04T16:33:38.864 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.864 DEBUG:boto:Data: 2024-03-04T16:33:38.864 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.864 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.865 DEBUG:boto:Params: {} 2024-03-04T16:33:38.865 DEBUG:boto:Token: None 2024-03-04T16:33:38.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e5738660739a99-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.868 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.868 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.868 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.868 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.868 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.868 DEBUG:boto:Data: 2024-03-04T16:33:38.868 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.868 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.869 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.869 DEBUG:boto:Params: {} 2024-03-04T16:33:38.869 DEBUG:boto:Token: None 2024-03-04T16:33:38.869 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.869 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f7c12f8f7943f1a-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.873 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.873 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.873 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.873 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:38.873 DEBUG:boto:Data: 2024-03-04T16:33:38.873 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.873 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.873 DEBUG:boto:Params: {} 2024-03-04T16:33:38.873 DEBUG:boto:Token: None 2024-03-04T16:33:38.874 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj 2024-03-04T16:33:38.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU= 2024-03-04T16:33:38.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dUIx7pq8IZxoDii2Yr7GdhRQxFU=', 'Content-Length': '0'} 2024-03-04T16:33:38.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f125bf7405e57d05-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.877 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.877 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.877 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.877 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.878 DEBUG:boto:Method: GET 2024-03-04T16:33:38.878 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.878 DEBUG:boto:Data: 2024-03-04T16:33:38.878 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.878 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.878 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.878 DEBUG:boto:Params: {} 2024-03-04T16:33:38.878 DEBUG:boto:Token: None 2024-03-04T16:33:38.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c022f4836d422d6-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.924 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.924 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.924 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.924 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.925 DEBUG:boto:Method: GET 2024-03-04T16:33:38.925 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.925 DEBUG:boto:Data: 2024-03-04T16:33:38.925 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.925 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.925 DEBUG:boto:Params: {} 2024-03-04T16:33:38.925 DEBUG:boto:Token: None 2024-03-04T16:33:38.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:38.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA= 2024-03-04T16:33:38.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sVokDuqYZj4TqBGwfph1yVCiWJA=', 'Content-Length': '0'} 2024-03-04T16:33:38.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000920a2f0d6d07607-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.928 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.928 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.928 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.928 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.928 DEBUG:boto:Method: GET 2024-03-04T16:33:38.929 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.929 DEBUG:boto:Data: 2024-03-04T16:33:38.929 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.929 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.929 DEBUG:boto:Params: {} 2024-03-04T16:33:38.929 DEBUG:boto:Token: None 2024-03-04T16:33:38.929 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.929 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e3f80627d38971-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.932 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.932 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:38.933 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:38.933 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.933 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:38.933 DEBUG:boto:Method: GET 2024-03-04T16:33:38.933 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:38.933 DEBUG:boto:Data: 2024-03-04T16:33:38.933 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.933 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.933 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.933 DEBUG:boto:Params: {} 2024-03-04T16:33:38.934 DEBUG:boto:Token: None 2024-03-04T16:33:38.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/ 2024-03-04T16:33:38.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss= 2024-03-04T16:33:38.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tOiOveUdV1Do0SHg4+iRQ9tA2Ss=', 'Content-Length': '0'} 2024-03-04T16:33:38.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b945751318ff5bb8-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.937 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:38.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.937 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.937 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.937 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.937 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.937 DEBUG:boto:Method: GET 2024-03-04T16:33:38.938 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.938 DEBUG:boto:Data: 2024-03-04T16:33:38.938 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.938 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.938 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.938 DEBUG:boto:Params: {} 2024-03-04T16:33:38.938 DEBUG:boto:Token: None 2024-03-04T16:33:38.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0= 2024-03-04T16:33:38.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0=', 'Content-Length': '0'} 2024-03-04T16:33:38.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8bce4fb4fca5201-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.941 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.941 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.941 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.941 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.941 DEBUG:boto:Method: GET 2024-03-04T16:33:38.941 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.941 DEBUG:boto:Data: 2024-03-04T16:33:38.941 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.942 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.942 DEBUG:boto:Params: {} 2024-03-04T16:33:38.942 DEBUG:boto:Token: None 2024-03-04T16:33:38.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0= 2024-03-04T16:33:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0=', 'Content-Length': '0'} 2024-03-04T16:33:38.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019239f78fee75f69-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.944 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.944 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.944 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.944 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.945 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.945 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.945 DEBUG:boto:Data: 2024-03-04T16:33:38.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.945 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.945 DEBUG:boto:Params: {} 2024-03-04T16:33:38.945 DEBUG:boto:Token: None 2024-03-04T16:33:38.945 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oH219pr9YKnILNJQ+01bixUYzUM= 2024-03-04T16:33:38.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oH219pr9YKnILNJQ+01bixUYzUM=', 'Content-Length': '0'} 2024-03-04T16:33:38.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000486bb5128ccc1fba-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.948 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.948 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.948 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.948 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.948 DEBUG:boto:Method: HEAD 2024-03-04T16:33:38.948 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.948 DEBUG:boto:Data: 2024-03-04T16:33:38.948 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.948 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.948 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.948 DEBUG:boto:Params: {} 2024-03-04T16:33:38.949 DEBUG:boto:Token: None 2024-03-04T16:33:38.949 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oH219pr9YKnILNJQ+01bixUYzUM= 2024-03-04T16:33:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oH219pr9YKnILNJQ+01bixUYzUM=', 'Content-Length': '0'} 2024-03-04T16:33:38.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f4bc0534f51bafb-0065e5f7e2-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:38.951 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.951 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.951 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.951 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.951 DEBUG:boto:Method: GET 2024-03-04T16:33:38.951 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.951 DEBUG:boto:Data: 2024-03-04T16:33:38.951 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.952 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:38.952 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.952 DEBUG:boto:Params: {} 2024-03-04T16:33:38.952 DEBUG:boto:Token: None 2024-03-04T16:33:38.952 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.952 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0= 2024-03-04T16:33:38.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0=', 'Content-Length': '0'} 2024-03-04T16:33:38.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fddbac52cab6fed-0065e5f7e2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:38.996 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:38.996 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:38.996 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.997 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.997 DEBUG:boto:Method: GET 2024-03-04T16:33:38.997 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.997 DEBUG:boto:Data: 2024-03-04T16:33:38.997 DEBUG:boto:Headers: {} 2024-03-04T16:33:38.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:38.997 DEBUG:boto:Port: 8000 2024-03-04T16:33:38.997 DEBUG:boto:Params: {} 2024-03-04T16:33:38.997 DEBUG:boto:Token: None 2024-03-04T16:33:38.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:38 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:38.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0= 2024-03-04T16:33:38.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q7OGuNJWbsaPQUxeeXQ1lwEJqv0=', 'Content-Length': '0'} 2024-03-04T16:33:39.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000500efc974f97e302-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.000 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.000 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.000 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.000 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.000 DEBUG:boto:Method: GET 2024-03-04T16:33:39.000 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.001 DEBUG:boto:Data: 2024-03-04T16:33:39.001 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.001 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.001 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.001 DEBUG:boto:Params: {} 2024-03-04T16:33:39.001 DEBUG:boto:Token: None 2024-03-04T16:33:39.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd5f1f89aadd103-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.004 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.005 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.005 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.005 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.005 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.005 DEBUG:boto:Method: GET 2024-03-04T16:33:39.005 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.005 DEBUG:boto:Data: 2024-03-04T16:33:39.005 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.006 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.006 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.006 DEBUG:boto:Params: {} 2024-03-04T16:33:39.006 DEBUG:boto:Token: None 2024-03-04T16:33:39.006 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.006 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b39866b8ea3cd3-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.009 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.009 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.009 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.010 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.010 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.010 DEBUG:boto:Data: 2024-03-04T16:33:39.010 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.010 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.010 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.010 DEBUG:boto:Params: {} 2024-03-04T16:33:39.010 DEBUG:boto:Token: None 2024-03-04T16:33:39.010 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026a379b152876b17-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.014 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.014 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.014 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.014 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.014 DEBUG:boto:Data: 2024-03-04T16:33:39.014 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.014 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.015 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.015 DEBUG:boto:Params: {} 2024-03-04T16:33:39.015 DEBUG:boto:Token: None 2024-03-04T16:33:39.015 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e47a6f35325000d0-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.019 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.019 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.019 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.019 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.019 DEBUG:boto:Method: GET 2024-03-04T16:33:39.019 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.019 DEBUG:boto:Data: 2024-03-04T16:33:39.019 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.019 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.019 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.019 DEBUG:boto:Params: {} 2024-03-04T16:33:39.019 DEBUG:boto:Token: None 2024-03-04T16:33:39.020 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca56521928f9e642-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.064 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.064 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.065 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.065 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.065 DEBUG:boto:Method: GET 2024-03-04T16:33:39.065 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.065 DEBUG:boto:Data: 2024-03-04T16:33:39.065 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.065 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.065 DEBUG:boto:Params: {} 2024-03-04T16:33:39.065 DEBUG:boto:Token: None 2024-03-04T16:33:39.066 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d5d033015851d4a-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.068 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.068 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.069 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.069 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.069 DEBUG:boto:Method: GET 2024-03-04T16:33:39.069 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.069 DEBUG:boto:Data: 2024-03-04T16:33:39.069 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.069 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.069 DEBUG:boto:Params: {} 2024-03-04T16:33:39.069 DEBUG:boto:Token: None 2024-03-04T16:33:39.070 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000640e48d9bb6a0a3e-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.073 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.073 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.074 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.074 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.074 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.074 DEBUG:boto:Method: GET 2024-03-04T16:33:39.074 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.074 DEBUG:boto:Data: 2024-03-04T16:33:39.074 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.074 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.075 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.075 DEBUG:boto:Params: {} 2024-03-04T16:33:39.075 DEBUG:boto:Token: None 2024-03-04T16:33:39.075 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.075 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000917e59b93671fb41-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.078 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.079 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.079 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.079 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.079 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.079 DEBUG:boto:Method: GET 2024-03-04T16:33:39.079 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.079 DEBUG:boto:Data: 2024-03-04T16:33:39.080 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.080 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.080 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.080 DEBUG:boto:Params: {} 2024-03-04T16:33:39.080 DEBUG:boto:Token: None 2024-03-04T16:33:39.080 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0= 2024-03-04T16:33:39.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0=', 'Content-Length': '0'} 2024-03-04T16:33:39.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3f7e785012554f2-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.083 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.083 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.083 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.083 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.083 DEBUG:boto:Method: GET 2024-03-04T16:33:39.083 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.083 DEBUG:boto:Data: 2024-03-04T16:33:39.083 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.083 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.083 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.083 DEBUG:boto:Params: {} 2024-03-04T16:33:39.084 DEBUG:boto:Token: None 2024-03-04T16:33:39.084 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.084 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0= 2024-03-04T16:33:39.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0=', 'Content-Length': '0'} 2024-03-04T16:33:39.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078a6d3b4be909552-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.128 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.128 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.128 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.129 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.129 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.129 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.129 DEBUG:boto:Data: 2024-03-04T16:33:39.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.129 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.129 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.129 DEBUG:boto:Params: {} 2024-03-04T16:33:39.129 DEBUG:boto:Token: None 2024-03-04T16:33:39.129 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GxK1VgL4VhiRj8Yphj7UEtH2PXw= 2024-03-04T16:33:39.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GxK1VgL4VhiRj8Yphj7UEtH2PXw=', 'Content-Length': '0'} 2024-03-04T16:33:39.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075679051eecb6e5f-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.132 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.132 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.132 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.132 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.132 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.133 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.133 DEBUG:boto:Data: 2024-03-04T16:33:39.133 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.133 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.133 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.133 DEBUG:boto:Params: {} 2024-03-04T16:33:39.133 DEBUG:boto:Token: None 2024-03-04T16:33:39.133 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GxK1VgL4VhiRj8Yphj7UEtH2PXw= 2024-03-04T16:33:39.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GxK1VgL4VhiRj8Yphj7UEtH2PXw=', 'Content-Length': '0'} 2024-03-04T16:33:39.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef55dc8e9eb27452-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.136 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.136 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.136 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.136 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.136 DEBUG:boto:Method: GET 2024-03-04T16:33:39.136 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.137 DEBUG:boto:Data: 2024-03-04T16:33:39.137 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.137 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.137 DEBUG:boto:Params: {} 2024-03-04T16:33:39.137 DEBUG:boto:Token: None 2024-03-04T16:33:39.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0= 2024-03-04T16:33:39.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0=', 'Content-Length': '0'} 2024-03-04T16:33:39.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf8564b61f1b51e5-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.140 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.140 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.140 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.140 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.140 DEBUG:boto:Method: GET 2024-03-04T16:33:39.140 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.140 DEBUG:boto:Data: 2024-03-04T16:33:39.140 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.140 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.141 DEBUG:boto:Params: {} 2024-03-04T16:33:39.141 DEBUG:boto:Token: None 2024-03-04T16:33:39.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:39.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0= 2024-03-04T16:33:39.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0SmVluN47dTnhVE+EE/PMEyS7S0=', 'Content-Length': '0'} 2024-03-04T16:33:39.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad774eddb1dae3a5-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.143 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.143 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.143 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.143 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.143 DEBUG:boto:Method: GET 2024-03-04T16:33:39.143 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.143 DEBUG:boto:Data: 2024-03-04T16:33:39.144 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.144 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.144 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.144 DEBUG:boto:Params: {} 2024-03-04T16:33:39.144 DEBUG:boto:Token: None 2024-03-04T16:33:39.144 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.144 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005320a4963ba9f2de-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.147 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.148 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.148 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.148 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.148 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.148 DEBUG:boto:Method: GET 2024-03-04T16:33:39.148 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.149 DEBUG:boto:Data: 2024-03-04T16:33:39.149 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.149 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.149 DEBUG:boto:Params: {} 2024-03-04T16:33:39.149 DEBUG:boto:Token: None 2024-03-04T16:33:39.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ff82c763cba653-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.152 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.153 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.153 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.153 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.153 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.153 DEBUG:boto:Data: 2024-03-04T16:33:39.153 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.153 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.153 DEBUG:boto:Params: {} 2024-03-04T16:33:39.153 DEBUG:boto:Token: None 2024-03-04T16:33:39.154 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c5114e3ad2321ca-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.158 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.158 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.158 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.158 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.158 DEBUG:boto:Data: 2024-03-04T16:33:39.158 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.159 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.159 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.159 DEBUG:boto:Params: {} 2024-03-04T16:33:39.159 DEBUG:boto:Token: None 2024-03-04T16:33:39.159 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bce4af329fc48cb0-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.163 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.163 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.163 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.164 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.164 DEBUG:boto:Method: GET 2024-03-04T16:33:39.164 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.164 DEBUG:boto:Data: 2024-03-04T16:33:39.164 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.164 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.164 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.164 DEBUG:boto:Params: {} 2024-03-04T16:33:39.164 DEBUG:boto:Token: None 2024-03-04T16:33:39.164 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e55c530a8b5811e-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.208 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.208 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.208 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.208 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.209 DEBUG:boto:Method: GET 2024-03-04T16:33:39.209 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.209 DEBUG:boto:Data: 2024-03-04T16:33:39.209 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.209 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.209 DEBUG:boto:Params: {} 2024-03-04T16:33:39.209 DEBUG:boto:Token: None 2024-03-04T16:33:39.209 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399885ce32901791-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.214 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.214 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.214 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.214 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.214 DEBUG:boto:Method: GET 2024-03-04T16:33:39.214 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.214 DEBUG:boto:Data: 2024-03-04T16:33:39.214 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.214 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.215 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.215 DEBUG:boto:Params: {} 2024-03-04T16:33:39.215 DEBUG:boto:Token: None 2024-03-04T16:33:39.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.215 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e429d7d45d8d9b73-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.218 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.219 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.219 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.219 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.219 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.220 DEBUG:boto:Method: GET 2024-03-04T16:33:39.220 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.220 DEBUG:boto:Data: 2024-03-04T16:33:39.220 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.220 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.220 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.220 DEBUG:boto:Params: {} 2024-03-04T16:33:39.220 DEBUG:boto:Token: None 2024-03-04T16:33:39.220 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.221 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a5ccbffbb426bf-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.264 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.265 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.265 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.265 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.265 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.265 DEBUG:boto:Method: GET 2024-03-04T16:33:39.265 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.265 DEBUG:boto:Data: 2024-03-04T16:33:39.265 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.265 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.265 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.266 DEBUG:boto:Params: {} 2024-03-04T16:33:39.266 DEBUG:boto:Token: None 2024-03-04T16:33:39.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s= 2024-03-04T16:33:39.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s=', 'Content-Length': '0'} 2024-03-04T16:33:39.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d8cae8f27fe545a-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.268 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.269 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.269 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.269 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.269 DEBUG:boto:Method: GET 2024-03-04T16:33:39.269 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.269 DEBUG:boto:Data: 2024-03-04T16:33:39.269 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.269 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.269 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.269 DEBUG:boto:Params: {} 2024-03-04T16:33:39.270 DEBUG:boto:Token: None 2024-03-04T16:33:39.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s= 2024-03-04T16:33:39.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s=', 'Content-Length': '0'} 2024-03-04T16:33:39.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ece1e54411a111d-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.272 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.272 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.272 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.272 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.272 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.273 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.273 DEBUG:boto:Data: 2024-03-04T16:33:39.273 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.273 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.273 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.273 DEBUG:boto:Params: {} 2024-03-04T16:33:39.273 DEBUG:boto:Token: None 2024-03-04T16:33:39.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.274 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:v7P4nsz8CTrlVCQbb8dmS1MK8i4= 2024-03-04T16:33:39.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v7P4nsz8CTrlVCQbb8dmS1MK8i4=', 'Content-Length': '0'} 2024-03-04T16:33:39.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091aa124c22888eec-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.276 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.276 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.276 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.276 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.276 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.276 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.276 DEBUG:boto:Data: 2024-03-04T16:33:39.276 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.277 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.277 DEBUG:boto:Params: {} 2024-03-04T16:33:39.277 DEBUG:boto:Token: None 2024-03-04T16:33:39.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:v7P4nsz8CTrlVCQbb8dmS1MK8i4= 2024-03-04T16:33:39.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v7P4nsz8CTrlVCQbb8dmS1MK8i4=', 'Content-Length': '0'} 2024-03-04T16:33:39.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017e23dc62d02c907-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.279 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.279 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.280 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.280 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.280 DEBUG:boto:Method: GET 2024-03-04T16:33:39.280 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.280 DEBUG:boto:Data: 2024-03-04T16:33:39.280 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.280 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.280 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.280 DEBUG:boto:Params: {} 2024-03-04T16:33:39.280 DEBUG:boto:Token: None 2024-03-04T16:33:39.281 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s= 2024-03-04T16:33:39.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s=', 'Content-Length': '0'} 2024-03-04T16:33:39.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e74864e4e7f9e1-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.283 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.283 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.283 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.283 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.283 DEBUG:boto:Method: GET 2024-03-04T16:33:39.283 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.284 DEBUG:boto:Data: 2024-03-04T16:33:39.284 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.284 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.284 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.284 DEBUG:boto:Params: {} 2024-03-04T16:33:39.284 DEBUG:boto:Token: None 2024-03-04T16:33:39.284 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:39.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s= 2024-03-04T16:33:39.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zt8+WZGjl9uF8Q9luwDcxKwC+9s=', 'Content-Length': '0'} 2024-03-04T16:33:39.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd3eb1aa490d70af-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.286 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.287 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.287 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.287 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.287 DEBUG:boto:Method: GET 2024-03-04T16:33:39.287 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.287 DEBUG:boto:Data: 2024-03-04T16:33:39.287 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.287 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.287 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.287 DEBUG:boto:Params: {} 2024-03-04T16:33:39.288 DEBUG:boto:Token: None 2024-03-04T16:33:39.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000df058b704df99d8-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.291 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.291 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.291 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.291 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.291 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.292 DEBUG:boto:Method: GET 2024-03-04T16:33:39.292 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.292 DEBUG:boto:Data: 2024-03-04T16:33:39.292 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.292 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.292 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.292 DEBUG:boto:Params: {} 2024-03-04T16:33:39.292 DEBUG:boto:Token: None 2024-03-04T16:33:39.292 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.293 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a6c774a8479e7f-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.295 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.296 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.296 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.296 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.296 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.296 DEBUG:boto:Data: 2024-03-04T16:33:39.296 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.297 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.297 DEBUG:boto:Params: {} 2024-03-04T16:33:39.297 DEBUG:boto:Token: None 2024-03-04T16:33:39.297 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.297 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d53bfe805696bddb-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.301 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.301 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.301 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.301 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.301 DEBUG:boto:Data: 2024-03-04T16:33:39.301 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.301 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.302 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.302 DEBUG:boto:Params: {} 2024-03-04T16:33:39.302 DEBUG:boto:Token: None 2024-03-04T16:33:39.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000adce508ad0ddabf-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.305 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.306 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.306 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.306 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.306 DEBUG:boto:Method: GET 2024-03-04T16:33:39.306 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.306 DEBUG:boto:Data: 2024-03-04T16:33:39.306 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.306 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.306 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.306 DEBUG:boto:Params: {} 2024-03-04T16:33:39.306 DEBUG:boto:Token: None 2024-03-04T16:33:39.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008200c333f81e59ec-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.356 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.356 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.356 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.357 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.357 DEBUG:boto:Method: GET 2024-03-04T16:33:39.357 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.357 DEBUG:boto:Data: 2024-03-04T16:33:39.357 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.357 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.357 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.357 DEBUG:boto:Params: {} 2024-03-04T16:33:39.357 DEBUG:boto:Token: None 2024-03-04T16:33:39.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.358 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070d47e1359f2281d-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.360 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.360 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.360 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.360 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.360 DEBUG:boto:Method: GET 2024-03-04T16:33:39.360 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.360 DEBUG:boto:Data: 2024-03-04T16:33:39.360 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.361 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.361 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.361 DEBUG:boto:Params: {} 2024-03-04T16:33:39.361 DEBUG:boto:Token: None 2024-03-04T16:33:39.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc81c0a12317c5db-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.364 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.365 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.366 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.366 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.366 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.366 DEBUG:boto:Method: GET 2024-03-04T16:33:39.366 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.366 DEBUG:boto:Data: 2024-03-04T16:33:39.366 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.366 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.366 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.366 DEBUG:boto:Params: {} 2024-03-04T16:33:39.366 DEBUG:boto:Token: None 2024-03-04T16:33:39.367 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.367 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017dc355614f2c6fb-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.370 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.371 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.371 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.371 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.371 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.371 DEBUG:boto:Method: GET 2024-03-04T16:33:39.371 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.371 DEBUG:boto:Data: 2024-03-04T16:33:39.371 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.371 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.371 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.372 DEBUG:boto:Params: {} 2024-03-04T16:33:39.372 DEBUG:boto:Token: None 2024-03-04T16:33:39.372 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.372 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA= 2024-03-04T16:33:39.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA=', 'Content-Length': '0'} 2024-03-04T16:33:39.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f29175e18d6242a1-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.420 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.420 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.420 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.420 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.421 DEBUG:boto:Method: GET 2024-03-04T16:33:39.421 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.421 DEBUG:boto:Data: 2024-03-04T16:33:39.421 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.421 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.421 DEBUG:boto:Params: {} 2024-03-04T16:33:39.421 DEBUG:boto:Token: None 2024-03-04T16:33:39.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA= 2024-03-04T16:33:39.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA=', 'Content-Length': '0'} 2024-03-04T16:33:39.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e8018d5ddbe1763-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.424 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.424 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.424 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.424 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.425 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.425 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.425 DEBUG:boto:Data: 2024-03-04T16:33:39.425 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.425 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.425 DEBUG:boto:Params: {} 2024-03-04T16:33:39.425 DEBUG:boto:Token: None 2024-03-04T16:33:39.426 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0bPPm2SxmidYmtBXhvVwEzfUyd0= 2024-03-04T16:33:39.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0bPPm2SxmidYmtBXhvVwEzfUyd0=', 'Content-Length': '0'} 2024-03-04T16:33:39.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae74485deea7c54e-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.428 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.428 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.428 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.428 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.428 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.428 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.428 DEBUG:boto:Data: 2024-03-04T16:33:39.428 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.428 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.428 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.429 DEBUG:boto:Params: {} 2024-03-04T16:33:39.429 DEBUG:boto:Token: None 2024-03-04T16:33:39.429 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0bPPm2SxmidYmtBXhvVwEzfUyd0= 2024-03-04T16:33:39.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0bPPm2SxmidYmtBXhvVwEzfUyd0=', 'Content-Length': '0'} 2024-03-04T16:33:39.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ce4922190da6596-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.431 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.431 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.431 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.431 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.431 DEBUG:boto:Method: GET 2024-03-04T16:33:39.431 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.431 DEBUG:boto:Data: 2024-03-04T16:33:39.432 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.432 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.432 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.432 DEBUG:boto:Params: {} 2024-03-04T16:33:39.432 DEBUG:boto:Token: None 2024-03-04T16:33:39.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA= 2024-03-04T16:33:39.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA=', 'Content-Length': '0'} 2024-03-04T16:33:39.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046a1b21a28e2188f-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.484 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.484 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.485 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.485 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.485 DEBUG:boto:Method: GET 2024-03-04T16:33:39.485 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.485 DEBUG:boto:Data: 2024-03-04T16:33:39.485 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.485 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.485 DEBUG:boto:Params: {} 2024-03-04T16:33:39.485 DEBUG:boto:Token: None 2024-03-04T16:33:39.486 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:39.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA= 2024-03-04T16:33:39.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JZppQwEM17GKYWc6XKhz4eLf2oA=', 'Content-Length': '0'} 2024-03-04T16:33:39.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4d44bec683abe89-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.488 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.488 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.488 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.488 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.488 DEBUG:boto:Method: GET 2024-03-04T16:33:39.488 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.488 DEBUG:boto:Data: 2024-03-04T16:33:39.488 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.488 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.489 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.489 DEBUG:boto:Params: {} 2024-03-04T16:33:39.489 DEBUG:boto:Token: None 2024-03-04T16:33:39.489 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0388e436129728d-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.532 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.533 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.533 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.533 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.533 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.533 DEBUG:boto:Method: GET 2024-03-04T16:33:39.533 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.533 DEBUG:boto:Data: 2024-03-04T16:33:39.533 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.533 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.533 DEBUG:boto:Params: {} 2024-03-04T16:33:39.534 DEBUG:boto:Token: None 2024-03-04T16:33:39.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004618ea6e032b5c4b-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.537 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.538 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.538 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.538 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.538 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.538 DEBUG:boto:Data: 2024-03-04T16:33:39.538 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.538 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.538 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.538 DEBUG:boto:Params: {} 2024-03-04T16:33:39.538 DEBUG:boto:Token: None 2024-03-04T16:33:39.539 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000509bc12f00e5d4f-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.542 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.542 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.542 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.542 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.542 DEBUG:boto:Data: 2024-03-04T16:33:39.542 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.542 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.543 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.543 DEBUG:boto:Params: {} 2024-03-04T16:33:39.543 DEBUG:boto:Token: None 2024-03-04T16:33:39.543 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d98bfae183ec87c-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.546 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.547 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.547 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.547 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.547 DEBUG:boto:Method: GET 2024-03-04T16:33:39.547 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.547 DEBUG:boto:Data: 2024-03-04T16:33:39.547 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.547 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.547 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.547 DEBUG:boto:Params: {} 2024-03-04T16:33:39.547 DEBUG:boto:Token: None 2024-03-04T16:33:39.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2c519b589241386-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.550 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.550 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.550 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.550 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.550 DEBUG:boto:Method: GET 2024-03-04T16:33:39.550 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.550 DEBUG:boto:Data: 2024-03-04T16:33:39.550 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.550 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.550 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.551 DEBUG:boto:Params: {} 2024-03-04T16:33:39.551 DEBUG:boto:Token: None 2024-03-04T16:33:39.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.551 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022972e6e5c765c92-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.596 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.600 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.600 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.600 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.600 DEBUG:boto:Method: GET 2024-03-04T16:33:39.600 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.600 DEBUG:boto:Data: 2024-03-04T16:33:39.601 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.601 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.601 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.601 DEBUG:boto:Params: {} 2024-03-04T16:33:39.601 DEBUG:boto:Token: None 2024-03-04T16:33:39.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005262a69acbc6d4e8-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.604 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.605 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.605 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.605 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.605 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.605 DEBUG:boto:Method: GET 2024-03-04T16:33:39.605 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.605 DEBUG:boto:Data: 2024-03-04T16:33:39.605 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.605 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.606 DEBUG:boto:Params: {} 2024-03-04T16:33:39.606 DEBUG:boto:Token: None 2024-03-04T16:33:39.606 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.606 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006729642c803b6c6c-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.610 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.610 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.610 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.611 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.611 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.611 DEBUG:boto:Method: GET 2024-03-04T16:33:39.611 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.611 DEBUG:boto:Data: 2024-03-04T16:33:39.611 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.611 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.611 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.611 DEBUG:boto:Params: {} 2024-03-04T16:33:39.611 DEBUG:boto:Token: None 2024-03-04T16:33:39.612 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.612 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ= 2024-03-04T16:33:39.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ=', 'Content-Length': '0'} 2024-03-04T16:33:39.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bba3f0f0a1b1c653-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.656 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.656 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.656 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.656 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.656 DEBUG:boto:Method: GET 2024-03-04T16:33:39.657 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.657 DEBUG:boto:Data: 2024-03-04T16:33:39.657 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.657 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.657 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.657 DEBUG:boto:Params: {} 2024-03-04T16:33:39.657 DEBUG:boto:Token: None 2024-03-04T16:33:39.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.657 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ= 2024-03-04T16:33:39.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ=', 'Content-Length': '0'} 2024-03-04T16:33:39.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089ec9bea72306a4d-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.660 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.660 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.660 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.660 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.660 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.660 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.660 DEBUG:boto:Data: 2024-03-04T16:33:39.660 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.660 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.661 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.661 DEBUG:boto:Params: {} 2024-03-04T16:33:39.661 DEBUG:boto:Token: None 2024-03-04T16:33:39.661 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.661 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oIkOoB2zTdGJnc9p+ZZrsaOV8Do= 2024-03-04T16:33:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oIkOoB2zTdGJnc9p+ZZrsaOV8Do=', 'Content-Length': '0'} 2024-03-04T16:33:39.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a60aed36168deb1d-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.663 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.663 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.663 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.663 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.663 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.663 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.663 DEBUG:boto:Data: 2024-03-04T16:33:39.664 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.664 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.664 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.664 DEBUG:boto:Params: {} 2024-03-04T16:33:39.664 DEBUG:boto:Token: None 2024-03-04T16:33:39.664 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.664 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oIkOoB2zTdGJnc9p+ZZrsaOV8Do= 2024-03-04T16:33:39.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oIkOoB2zTdGJnc9p+ZZrsaOV8Do=', 'Content-Length': '0'} 2024-03-04T16:33:39.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec185aa9b572b5cd-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.667 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.667 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.667 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.667 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.667 DEBUG:boto:Method: GET 2024-03-04T16:33:39.667 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.667 DEBUG:boto:Data: 2024-03-04T16:33:39.667 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.667 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.667 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.667 DEBUG:boto:Params: {} 2024-03-04T16:33:39.668 DEBUG:boto:Token: None 2024-03-04T16:33:39.668 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.668 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ= 2024-03-04T16:33:39.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ=', 'Content-Length': '0'} 2024-03-04T16:33:39.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a81db5e6ccae41c2-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.712 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.712 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.712 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.712 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.713 DEBUG:boto:Method: GET 2024-03-04T16:33:39.713 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.713 DEBUG:boto:Data: 2024-03-04T16:33:39.713 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.713 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.713 DEBUG:boto:Params: {} 2024-03-04T16:33:39.713 DEBUG:boto:Token: None 2024-03-04T16:33:39.713 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:39.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ= 2024-03-04T16:33:39.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rKOkvHXyVZiTWBzzOMrfCsJyemQ=', 'Content-Length': '0'} 2024-03-04T16:33:39.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f97cb2efeda05b6b-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.716 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.716 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.716 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.716 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.716 DEBUG:boto:Method: GET 2024-03-04T16:33:39.716 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.716 DEBUG:boto:Data: 2024-03-04T16:33:39.717 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.717 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.717 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.717 DEBUG:boto:Params: {} 2024-03-04T16:33:39.717 DEBUG:boto:Token: None 2024-03-04T16:33:39.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.717 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009cb947f01d63568-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.720 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.720 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.720 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.720 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.720 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.720 DEBUG:boto:Method: GET 2024-03-04T16:33:39.721 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.721 DEBUG:boto:Data: 2024-03-04T16:33:39.721 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.721 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.721 DEBUG:boto:Params: {} 2024-03-04T16:33:39.721 DEBUG:boto:Token: None 2024-03-04T16:33:39.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be07085c87524eb-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.724 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.725 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.725 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.725 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.725 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.725 DEBUG:boto:Data: 2024-03-04T16:33:39.725 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.726 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.726 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.726 DEBUG:boto:Params: {} 2024-03-04T16:33:39.726 DEBUG:boto:Token: None 2024-03-04T16:33:39.726 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d62c967db9e96699-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.729 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.729 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.729 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.729 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.730 DEBUG:boto:Data: 2024-03-04T16:33:39.730 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.730 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.730 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.730 DEBUG:boto:Params: {} 2024-03-04T16:33:39.730 DEBUG:boto:Token: None 2024-03-04T16:33:39.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b35dd015da9c2d88-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.735 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.735 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.735 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.735 DEBUG:boto:Method: GET 2024-03-04T16:33:39.735 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.735 DEBUG:boto:Data: 2024-03-04T16:33:39.735 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.735 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.735 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.736 DEBUG:boto:Params: {} 2024-03-04T16:33:39.736 DEBUG:boto:Token: None 2024-03-04T16:33:39.736 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000461a51b73b6d9266-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.739 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.739 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.739 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.739 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.739 DEBUG:boto:Method: GET 2024-03-04T16:33:39.739 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.739 DEBUG:boto:Data: 2024-03-04T16:33:39.739 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.739 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.740 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.740 DEBUG:boto:Params: {} 2024-03-04T16:33:39.740 DEBUG:boto:Token: None 2024-03-04T16:33:39.740 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.740 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e786df09712214de-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.788 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.788 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.788 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.789 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.789 DEBUG:boto:Method: GET 2024-03-04T16:33:39.789 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.789 DEBUG:boto:Data: 2024-03-04T16:33:39.789 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.789 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.789 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.789 DEBUG:boto:Params: {} 2024-03-04T16:33:39.789 DEBUG:boto:Token: None 2024-03-04T16:33:39.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000398629ba48233db6-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.793 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.793 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.793 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.793 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.793 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.793 DEBUG:boto:Method: GET 2024-03-04T16:33:39.794 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.794 DEBUG:boto:Data: 2024-03-04T16:33:39.794 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.794 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.794 DEBUG:boto:Params: {} 2024-03-04T16:33:39.794 DEBUG:boto:Token: None 2024-03-04T16:33:39.794 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b38827b0944705-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.798 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.798 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.798 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.798 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.799 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.799 DEBUG:boto:Method: GET 2024-03-04T16:33:39.799 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.799 DEBUG:boto:Data: 2024-03-04T16:33:39.799 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.799 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.799 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.799 DEBUG:boto:Params: {} 2024-03-04T16:33:39.799 DEBUG:boto:Token: None 2024-03-04T16:33:39.800 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.800 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0= 2024-03-04T16:33:39.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0=', 'Content-Length': '0'} 2024-03-04T16:33:39.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fd6fc4d9741ef64-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.848 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.848 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.848 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.849 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.849 DEBUG:boto:Method: GET 2024-03-04T16:33:39.849 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.849 DEBUG:boto:Data: 2024-03-04T16:33:39.849 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.849 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.849 DEBUG:boto:Params: {} 2024-03-04T16:33:39.849 DEBUG:boto:Token: None 2024-03-04T16:33:39.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0= 2024-03-04T16:33:39.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0=', 'Content-Length': '0'} 2024-03-04T16:33:39.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb493e811e42379-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.852 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.853 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.853 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.853 DEBUG:boto:Data: 2024-03-04T16:33:39.853 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.853 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.853 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.853 DEBUG:boto:Params: {} 2024-03-04T16:33:39.853 DEBUG:boto:Token: None 2024-03-04T16:33:39.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.854 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Dd0Ian+LwmBpkmmWJhLijngvzl4= 2024-03-04T16:33:39.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Dd0Ian+LwmBpkmmWJhLijngvzl4=', 'Content-Length': '0'} 2024-03-04T16:33:39.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008946ae62d9ad2852-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.856 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.856 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.856 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.856 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.856 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.856 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.856 DEBUG:boto:Data: 2024-03-04T16:33:39.856 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.857 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.857 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.857 DEBUG:boto:Params: {} 2024-03-04T16:33:39.857 DEBUG:boto:Token: None 2024-03-04T16:33:39.857 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Dd0Ian+LwmBpkmmWJhLijngvzl4= 2024-03-04T16:33:39.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Dd0Ian+LwmBpkmmWJhLijngvzl4=', 'Content-Length': '0'} 2024-03-04T16:33:39.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5a37ade2924ec44-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.860 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.860 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.860 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.860 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.860 DEBUG:boto:Method: GET 2024-03-04T16:33:39.860 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.860 DEBUG:boto:Data: 2024-03-04T16:33:39.860 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.860 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.860 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.861 DEBUG:boto:Params: {} 2024-03-04T16:33:39.861 DEBUG:boto:Token: None 2024-03-04T16:33:39.861 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.861 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0= 2024-03-04T16:33:39.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0=', 'Content-Length': '0'} 2024-03-04T16:33:39.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f38e013e11bc5ed-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.863 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.863 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.863 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.863 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.864 DEBUG:boto:Method: GET 2024-03-04T16:33:39.864 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.864 DEBUG:boto:Data: 2024-03-04T16:33:39.864 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.864 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.864 DEBUG:boto:Params: {} 2024-03-04T16:33:39.864 DEBUG:boto:Token: None 2024-03-04T16:33:39.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:39.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0= 2024-03-04T16:33:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tw6frfd1oyfxje3ZRxjlHfy2eI0=', 'Content-Length': '0'} 2024-03-04T16:33:39.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb4000bac0e368d9-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.867 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.867 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.867 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.867 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.868 DEBUG:boto:Method: GET 2024-03-04T16:33:39.868 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.868 DEBUG:boto:Data: 2024-03-04T16:33:39.868 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.868 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.868 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.868 DEBUG:boto:Params: {} 2024-03-04T16:33:39.868 DEBUG:boto:Token: None 2024-03-04T16:33:39.868 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.869 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a1391660d48008-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.871 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.871 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.872 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.872 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.872 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.872 DEBUG:boto:Method: GET 2024-03-04T16:33:39.872 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.872 DEBUG:boto:Data: 2024-03-04T16:33:39.872 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.872 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.872 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.873 DEBUG:boto:Params: {} 2024-03-04T16:33:39.873 DEBUG:boto:Token: None 2024-03-04T16:33:39.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.873 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e57fe9f0cbd0272-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.876 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.876 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.876 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.877 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.877 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.877 DEBUG:boto:Data: 2024-03-04T16:33:39.877 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.877 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.877 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.877 DEBUG:boto:Params: {} 2024-03-04T16:33:39.877 DEBUG:boto:Token: None 2024-03-04T16:33:39.877 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.877 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e416ea0ba86745eb-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.881 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.881 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.881 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.881 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:39.881 DEBUG:boto:Data: 2024-03-04T16:33:39.881 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.881 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.881 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.881 DEBUG:boto:Params: {} 2024-03-04T16:33:39.882 DEBUG:boto:Token: None 2024-03-04T16:33:39.882 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj 2024-03-04T16:33:39.882 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k= 2024-03-04T16:33:39.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:K8IWjvm2U5pU3lQOHGYE9cLE73k=', 'Content-Length': '0'} 2024-03-04T16:33:39.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ac90a259e4ebe1c-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.886 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.886 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.886 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.886 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.886 DEBUG:boto:Method: GET 2024-03-04T16:33:39.886 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.886 DEBUG:boto:Data: 2024-03-04T16:33:39.887 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.887 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.887 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.887 DEBUG:boto:Params: {} 2024-03-04T16:33:39.887 DEBUG:boto:Token: None 2024-03-04T16:33:39.887 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.887 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea6ecfab78508cad-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.889 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.890 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.890 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.890 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.890 DEBUG:boto:Method: GET 2024-03-04T16:33:39.890 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.890 DEBUG:boto:Data: 2024-03-04T16:33:39.890 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.890 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.890 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.891 DEBUG:boto:Params: {} 2024-03-04T16:33:39.891 DEBUG:boto:Token: None 2024-03-04T16:33:39.891 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:39.891 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw= 2024-03-04T16:33:39.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YVMcBwSBtcvJmKr4dj0rfyTXzZw=', 'Content-Length': '0'} 2024-03-04T16:33:39.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea295a925950e82c-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.936 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.936 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.936 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.936 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.937 DEBUG:boto:Method: GET 2024-03-04T16:33:39.937 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.937 DEBUG:boto:Data: 2024-03-04T16:33:39.937 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.937 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.937 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.937 DEBUG:boto:Params: {} 2024-03-04T16:33:39.937 DEBUG:boto:Token: None 2024-03-04T16:33:39.937 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb48ce36e49594c9-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.940 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.941 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:39.941 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:39.941 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.941 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:39.941 DEBUG:boto:Method: GET 2024-03-04T16:33:39.941 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:39.941 DEBUG:boto:Data: 2024-03-04T16:33:39.941 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.942 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.942 DEBUG:boto:Params: {} 2024-03-04T16:33:39.942 DEBUG:boto:Token: None 2024-03-04T16:33:39.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/ 2024-03-04T16:33:39.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM= 2024-03-04T16:33:39.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Nw0fMIeh50GqDjUzX/RE1OvYJdM=', 'Content-Length': '0'} 2024-03-04T16:33:39.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc396bc9eac4e85c-0065e5f7e3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.945 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:39.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:39.945 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.946 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.946 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.946 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.946 DEBUG:boto:Method: GET 2024-03-04T16:33:39.946 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.946 DEBUG:boto:Data: 2024-03-04T16:33:39.946 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.946 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.946 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.946 DEBUG:boto:Params: {} 2024-03-04T16:33:39.946 DEBUG:boto:Token: None 2024-03-04T16:33:39.947 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.947 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hRk1MGV+zlLxnVRwAfq5pLWBBWg= 2024-03-04T16:33:39.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hRk1MGV+zlLxnVRwAfq5pLWBBWg=', 'Content-Length': '0'} 2024-03-04T16:33:39.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b316ab565e478fa-0065e5f7e3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.992 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.992 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.992 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.993 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.993 DEBUG:boto:Method: GET 2024-03-04T16:33:39.993 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.993 DEBUG:boto:Data: 2024-03-04T16:33:39.993 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:39.993 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.993 DEBUG:boto:Params: {} 2024-03-04T16:33:39.993 DEBUG:boto:Token: None 2024-03-04T16:33:39.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hRk1MGV+zlLxnVRwAfq5pLWBBWg= 2024-03-04T16:33:39.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hRk1MGV+zlLxnVRwAfq5pLWBBWg=', 'Content-Length': '0'} 2024-03-04T16:33:39.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2fd7876d1eefc4-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:39.996 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:39.996 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:39.996 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.997 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.997 DEBUG:boto:Method: HEAD 2024-03-04T16:33:39.997 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.997 DEBUG:boto:Data: 2024-03-04T16:33:39.997 DEBUG:boto:Headers: {} 2024-03-04T16:33:39.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:39.997 DEBUG:boto:Port: 8000 2024-03-04T16:33:39.997 DEBUG:boto:Params: {} 2024-03-04T16:33:39.997 DEBUG:boto:Token: None 2024-03-04T16:33:39.997 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:39 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:39.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2iOJAu7IG+kylhkfINwMz/oh2Zc= 2024-03-04T16:33:39.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2iOJAu7IG+kylhkfINwMz/oh2Zc=', 'Content-Length': '0'} 2024-03-04T16:33:39.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be616a3861866e4d-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.000 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.000 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.000 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.000 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.000 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.000 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.000 DEBUG:boto:Data: 2024-03-04T16:33:40.000 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.000 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.000 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.001 DEBUG:boto:Params: {} 2024-03-04T16:33:40.001 DEBUG:boto:Token: None 2024-03-04T16:33:40.001 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z8Q/C9lI/F/E/YgnV3qiVMg7JVk= 2024-03-04T16:33:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z8Q/C9lI/F/E/YgnV3qiVMg7JVk=', 'Content-Length': '0'} 2024-03-04T16:33:40.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c38acc008c87daf-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.003 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.003 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.004 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.004 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.004 DEBUG:boto:Method: GET 2024-03-04T16:33:40.004 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.004 DEBUG:boto:Data: 2024-03-04T16:33:40.004 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.004 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.004 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.004 DEBUG:boto:Params: {} 2024-03-04T16:33:40.004 DEBUG:boto:Token: None 2024-03-04T16:33:40.004 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.005 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k1QcHZOAstqKle9d0ae4BwIZunU= 2024-03-04T16:33:40.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k1QcHZOAstqKle9d0ae4BwIZunU=', 'Content-Length': '0'} 2024-03-04T16:33:40.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012aedf1fe2ba3410-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.056 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.056 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.056 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.056 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.057 DEBUG:boto:Method: GET 2024-03-04T16:33:40.057 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.057 DEBUG:boto:Data: 2024-03-04T16:33:40.057 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.057 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.057 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.057 DEBUG:boto:Params: {} 2024-03-04T16:33:40.057 DEBUG:boto:Token: None 2024-03-04T16:33:40.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:40.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k1QcHZOAstqKle9d0ae4BwIZunU= 2024-03-04T16:33:40.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k1QcHZOAstqKle9d0ae4BwIZunU=', 'Content-Length': '0'} 2024-03-04T16:33:40.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088557182a1caee9b-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.060 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.060 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.060 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.060 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.060 DEBUG:boto:Method: GET 2024-03-04T16:33:40.061 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.061 DEBUG:boto:Data: 2024-03-04T16:33:40.061 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.061 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.061 DEBUG:boto:Params: {} 2024-03-04T16:33:40.061 DEBUG:boto:Token: None 2024-03-04T16:33:40.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e32f0552bb754f8-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.064 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.064 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.064 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.065 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.065 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.065 DEBUG:boto:Method: GET 2024-03-04T16:33:40.065 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.065 DEBUG:boto:Data: 2024-03-04T16:33:40.065 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.065 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.065 DEBUG:boto:Params: {} 2024-03-04T16:33:40.065 DEBUG:boto:Token: None 2024-03-04T16:33:40.065 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c446d4487b1dd046-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.068 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.069 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.069 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.069 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.069 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.069 DEBUG:boto:Data: 2024-03-04T16:33:40.069 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.069 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.069 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.070 DEBUG:boto:Params: {} 2024-03-04T16:33:40.070 DEBUG:boto:Token: None 2024-03-04T16:33:40.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061f8debb2bebbfca-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.073 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.073 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.073 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.073 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.074 DEBUG:boto:Data: 2024-03-04T16:33:40.074 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.074 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.074 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.074 DEBUG:boto:Params: {} 2024-03-04T16:33:40.074 DEBUG:boto:Token: None 2024-03-04T16:33:40.074 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d47ae9ddd1379e9c-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.078 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.078 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.078 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.078 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.078 DEBUG:boto:Method: GET 2024-03-04T16:33:40.079 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.079 DEBUG:boto:Data: 2024-03-04T16:33:40.079 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.079 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.079 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.079 DEBUG:boto:Params: {} 2024-03-04T16:33:40.079 DEBUG:boto:Token: None 2024-03-04T16:33:40.079 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.079 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000357df04ee8609e00-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.128 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.128 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.128 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.128 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.129 DEBUG:boto:Method: GET 2024-03-04T16:33:40.129 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.129 DEBUG:boto:Data: 2024-03-04T16:33:40.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.129 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.129 DEBUG:boto:Params: {} 2024-03-04T16:33:40.129 DEBUG:boto:Token: None 2024-03-04T16:33:40.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eacd78648403fd88-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.134 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.134 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.134 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.134 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.134 DEBUG:boto:Method: GET 2024-03-04T16:33:40.134 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.135 DEBUG:boto:Data: 2024-03-04T16:33:40.135 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.135 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.135 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.135 DEBUG:boto:Params: {} 2024-03-04T16:33:40.135 DEBUG:boto:Token: None 2024-03-04T16:33:40.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001074cdda071a618b-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.138 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.138 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.139 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.139 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.139 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.139 DEBUG:boto:Method: GET 2024-03-04T16:33:40.139 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.139 DEBUG:boto:Data: 2024-03-04T16:33:40.139 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.139 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.139 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.139 DEBUG:boto:Params: {} 2024-03-04T16:33:40.139 DEBUG:boto:Token: None 2024-03-04T16:33:40.140 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.140 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbedff3f5c6cb8ef-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.142 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.143 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.143 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.143 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.143 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.143 DEBUG:boto:Method: GET 2024-03-04T16:33:40.143 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.143 DEBUG:boto:Data: 2024-03-04T16:33:40.144 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.144 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.144 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.144 DEBUG:boto:Params: {} 2024-03-04T16:33:40.144 DEBUG:boto:Token: None 2024-03-04T16:33:40.144 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.144 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg= 2024-03-04T16:33:40.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg=', 'Content-Length': '0'} 2024-03-04T16:33:40.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f9d778337c511e-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.188 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.188 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.188 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.188 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.189 DEBUG:boto:Method: GET 2024-03-04T16:33:40.189 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.189 DEBUG:boto:Data: 2024-03-04T16:33:40.189 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.189 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.189 DEBUG:boto:Params: {} 2024-03-04T16:33:40.189 DEBUG:boto:Token: None 2024-03-04T16:33:40.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg= 2024-03-04T16:33:40.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg=', 'Content-Length': '0'} 2024-03-04T16:33:40.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007be962f53f1c02cf-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.192 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.192 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.192 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.192 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.192 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.193 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.193 DEBUG:boto:Data: 2024-03-04T16:33:40.193 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.193 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.193 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.193 DEBUG:boto:Params: {} 2024-03-04T16:33:40.193 DEBUG:boto:Token: None 2024-03-04T16:33:40.193 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.193 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fU+kXliWfqhygiD1Cqm1dEiYcx4= 2024-03-04T16:33:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fU+kXliWfqhygiD1Cqm1dEiYcx4=', 'Content-Length': '0'} 2024-03-04T16:33:40.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e6df2446ec165c0-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.196 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.196 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.196 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.196 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.196 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.196 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.196 DEBUG:boto:Data: 2024-03-04T16:33:40.197 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.197 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.197 DEBUG:boto:Params: {} 2024-03-04T16:33:40.197 DEBUG:boto:Token: None 2024-03-04T16:33:40.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fU+kXliWfqhygiD1Cqm1dEiYcx4= 2024-03-04T16:33:40.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fU+kXliWfqhygiD1Cqm1dEiYcx4=', 'Content-Length': '0'} 2024-03-04T16:33:40.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f13b624f1e9f3ff-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.200 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.200 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.201 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.201 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.201 DEBUG:boto:Method: GET 2024-03-04T16:33:40.201 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.201 DEBUG:boto:Data: 2024-03-04T16:33:40.201 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.201 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.201 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.201 DEBUG:boto:Params: {} 2024-03-04T16:33:40.201 DEBUG:boto:Token: None 2024-03-04T16:33:40.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg= 2024-03-04T16:33:40.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg=', 'Content-Length': '0'} 2024-03-04T16:33:40.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012e796caf91bccb8-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.244 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.244 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.244 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.245 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.245 DEBUG:boto:Method: GET 2024-03-04T16:33:40.245 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.245 DEBUG:boto:Data: 2024-03-04T16:33:40.245 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.245 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.245 DEBUG:boto:Params: {} 2024-03-04T16:33:40.245 DEBUG:boto:Token: None 2024-03-04T16:33:40.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:40.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg= 2024-03-04T16:33:40.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g8sRn9u8JlC5hQz4wD2WY8cZefg=', 'Content-Length': '0'} 2024-03-04T16:33:40.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1a42511d86db09b-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.248 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.248 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.248 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.248 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.248 DEBUG:boto:Method: GET 2024-03-04T16:33:40.249 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.249 DEBUG:boto:Data: 2024-03-04T16:33:40.249 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.249 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.249 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.249 DEBUG:boto:Params: {} 2024-03-04T16:33:40.249 DEBUG:boto:Token: None 2024-03-04T16:33:40.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3af675bb29bc0b0-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.296 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.297 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.297 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.297 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.297 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.297 DEBUG:boto:Method: GET 2024-03-04T16:33:40.297 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.298 DEBUG:boto:Data: 2024-03-04T16:33:40.298 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.298 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.298 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.298 DEBUG:boto:Params: {} 2024-03-04T16:33:40.298 DEBUG:boto:Token: None 2024-03-04T16:33:40.298 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.299 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f142d7e2f1efeb9-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.303 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.303 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.303 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.303 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.303 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.304 DEBUG:boto:Data: 2024-03-04T16:33:40.304 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.304 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.304 DEBUG:boto:Params: {} 2024-03-04T16:33:40.304 DEBUG:boto:Token: None 2024-03-04T16:33:40.304 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.304 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015b0b05e2f53aeb7-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.308 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.308 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.308 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.308 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.308 DEBUG:boto:Data: 2024-03-04T16:33:40.308 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.308 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.308 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.308 DEBUG:boto:Params: {} 2024-03-04T16:33:40.309 DEBUG:boto:Token: None 2024-03-04T16:33:40.309 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.309 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325b8d9a67edb51c-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.312 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.313 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.313 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.313 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.313 DEBUG:boto:Method: GET 2024-03-04T16:33:40.313 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.313 DEBUG:boto:Data: 2024-03-04T16:33:40.313 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.313 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.313 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.313 DEBUG:boto:Params: {} 2024-03-04T16:33:40.313 DEBUG:boto:Token: None 2024-03-04T16:33:40.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000932dc74578368d19-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.316 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.316 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.316 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.316 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.316 DEBUG:boto:Method: GET 2024-03-04T16:33:40.316 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.316 DEBUG:boto:Data: 2024-03-04T16:33:40.317 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.317 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.317 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.317 DEBUG:boto:Params: {} 2024-03-04T16:33:40.317 DEBUG:boto:Token: None 2024-03-04T16:33:40.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.317 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d310e0cca607f6a-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.319 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.319 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.319 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.320 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.320 DEBUG:boto:Method: GET 2024-03-04T16:33:40.320 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.320 DEBUG:boto:Data: 2024-03-04T16:33:40.320 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.320 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.320 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.320 DEBUG:boto:Params: {} 2024-03-04T16:33:40.320 DEBUG:boto:Token: None 2024-03-04T16:33:40.320 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.321 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40e02fd37986240-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.323 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.324 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.324 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.324 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.324 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.324 DEBUG:boto:Method: GET 2024-03-04T16:33:40.324 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.325 DEBUG:boto:Data: 2024-03-04T16:33:40.325 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.325 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.325 DEBUG:boto:Params: {} 2024-03-04T16:33:40.325 DEBUG:boto:Token: None 2024-03-04T16:33:40.325 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.325 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000baf04116fac6f31-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.328 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.328 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.329 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.329 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.329 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.329 DEBUG:boto:Method: GET 2024-03-04T16:33:40.329 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.329 DEBUG:boto:Data: 2024-03-04T16:33:40.329 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.329 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.329 DEBUG:boto:Params: {} 2024-03-04T16:33:40.329 DEBUG:boto:Token: None 2024-03-04T16:33:40.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68= 2024-03-04T16:33:40.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68=', 'Content-Length': '0'} 2024-03-04T16:33:40.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e91cff7d3925648-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.380 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.380 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.380 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.380 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.381 DEBUG:boto:Method: GET 2024-03-04T16:33:40.381 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.381 DEBUG:boto:Data: 2024-03-04T16:33:40.381 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.381 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.381 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.381 DEBUG:boto:Params: {} 2024-03-04T16:33:40.381 DEBUG:boto:Token: None 2024-03-04T16:33:40.381 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68= 2024-03-04T16:33:40.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68=', 'Content-Length': '0'} 2024-03-04T16:33:40.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000967f5bdcacd50acc-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.385 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.385 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.385 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.385 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.385 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.385 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.385 DEBUG:boto:Data: 2024-03-04T16:33:40.385 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.386 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.386 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.386 DEBUG:boto:Params: {} 2024-03-04T16:33:40.386 DEBUG:boto:Token: None 2024-03-04T16:33:40.386 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.386 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YCVTX0FIrtAINLlZSKYzoJUZifU= 2024-03-04T16:33:40.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YCVTX0FIrtAINLlZSKYzoJUZifU=', 'Content-Length': '0'} 2024-03-04T16:33:40.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6f316ff270bdb25-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.388 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.388 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.388 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.389 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.389 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.389 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.389 DEBUG:boto:Data: 2024-03-04T16:33:40.389 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.389 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.389 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.389 DEBUG:boto:Params: {} 2024-03-04T16:33:40.389 DEBUG:boto:Token: None 2024-03-04T16:33:40.389 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.389 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YCVTX0FIrtAINLlZSKYzoJUZifU= 2024-03-04T16:33:40.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YCVTX0FIrtAINLlZSKYzoJUZifU=', 'Content-Length': '0'} 2024-03-04T16:33:40.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca6fff425401ef36-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.392 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.392 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.392 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.392 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.392 DEBUG:boto:Method: GET 2024-03-04T16:33:40.392 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.392 DEBUG:boto:Data: 2024-03-04T16:33:40.392 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.392 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.392 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.393 DEBUG:boto:Params: {} 2024-03-04T16:33:40.393 DEBUG:boto:Token: None 2024-03-04T16:33:40.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.393 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68= 2024-03-04T16:33:40.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68=', 'Content-Length': '0'} 2024-03-04T16:33:40.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c0cd0198a338f11-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.440 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.440 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.441 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.441 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.441 DEBUG:boto:Method: GET 2024-03-04T16:33:40.441 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.441 DEBUG:boto:Data: 2024-03-04T16:33:40.441 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.441 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.441 DEBUG:boto:Params: {} 2024-03-04T16:33:40.441 DEBUG:boto:Token: None 2024-03-04T16:33:40.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:40.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68= 2024-03-04T16:33:40.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IL5R1WnAXSo1UUZGf6VknKYTX68=', 'Content-Length': '0'} 2024-03-04T16:33:40.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bf05aa931e066cf-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.451 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.451 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.451 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.452 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.452 DEBUG:boto:Method: GET 2024-03-04T16:33:40.452 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.452 DEBUG:boto:Data: 2024-03-04T16:33:40.452 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.452 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.452 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.452 DEBUG:boto:Params: {} 2024-03-04T16:33:40.452 DEBUG:boto:Token: None 2024-03-04T16:33:40.452 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd2193297a22a05-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.455 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.456 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.456 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.456 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.456 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.456 DEBUG:boto:Method: GET 2024-03-04T16:33:40.456 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.456 DEBUG:boto:Data: 2024-03-04T16:33:40.457 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.457 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.457 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.457 DEBUG:boto:Params: {} 2024-03-04T16:33:40.457 DEBUG:boto:Token: None 2024-03-04T16:33:40.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e8dd18627382e6f-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.474 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.474 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.474 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.474 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.474 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.474 DEBUG:boto:Data: 2024-03-04T16:33:40.475 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.475 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.475 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.475 DEBUG:boto:Params: {} 2024-03-04T16:33:40.475 DEBUG:boto:Token: None 2024-03-04T16:33:40.475 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.475 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044f4fc1b4bd3fe21-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.479 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.479 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.479 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.479 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.479 DEBUG:boto:Data: 2024-03-04T16:33:40.479 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.479 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.479 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.479 DEBUG:boto:Params: {} 2024-03-04T16:33:40.480 DEBUG:boto:Token: None 2024-03-04T16:33:40.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.480 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058ca04b78ad0a536-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.483 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.483 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.483 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.483 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.483 DEBUG:boto:Method: GET 2024-03-04T16:33:40.483 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.483 DEBUG:boto:Data: 2024-03-04T16:33:40.484 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.484 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.484 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.484 DEBUG:boto:Params: {} 2024-03-04T16:33:40.484 DEBUG:boto:Token: None 2024-03-04T16:33:40.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.484 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef45de456af39828-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.486 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.486 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.486 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.486 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.487 DEBUG:boto:Method: GET 2024-03-04T16:33:40.487 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.487 DEBUG:boto:Data: 2024-03-04T16:33:40.487 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.487 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.487 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.487 DEBUG:boto:Params: {} 2024-03-04T16:33:40.487 DEBUG:boto:Token: None 2024-03-04T16:33:40.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c4ddbf586119abd-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.536 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.536 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.536 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.537 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.537 DEBUG:boto:Method: GET 2024-03-04T16:33:40.537 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.537 DEBUG:boto:Data: 2024-03-04T16:33:40.537 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.537 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.537 DEBUG:boto:Params: {} 2024-03-04T16:33:40.537 DEBUG:boto:Token: None 2024-03-04T16:33:40.537 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8b16a742cfc503-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.541 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.541 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.541 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.541 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.541 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.541 DEBUG:boto:Method: GET 2024-03-04T16:33:40.541 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.542 DEBUG:boto:Data: 2024-03-04T16:33:40.542 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.542 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.542 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.542 DEBUG:boto:Params: {} 2024-03-04T16:33:40.542 DEBUG:boto:Token: None 2024-03-04T16:33:40.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ef1ae6d9e35e9e-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.545 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.546 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.546 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.546 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.546 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.546 DEBUG:boto:Method: GET 2024-03-04T16:33:40.546 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.546 DEBUG:boto:Data: 2024-03-04T16:33:40.546 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.546 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.546 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.547 DEBUG:boto:Params: {} 2024-03-04T16:33:40.547 DEBUG:boto:Token: None 2024-03-04T16:33:40.547 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.547 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo= 2024-03-04T16:33:40.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo=', 'Content-Length': '0'} 2024-03-04T16:33:40.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c49bef5da09363af-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.592 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.592 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.592 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.593 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.593 DEBUG:boto:Method: GET 2024-03-04T16:33:40.593 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.593 DEBUG:boto:Data: 2024-03-04T16:33:40.593 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.593 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.593 DEBUG:boto:Params: {} 2024-03-04T16:33:40.593 DEBUG:boto:Token: None 2024-03-04T16:33:40.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo= 2024-03-04T16:33:40.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo=', 'Content-Length': '0'} 2024-03-04T16:33:40.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d06678e5ad7fc917-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.596 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.596 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.596 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.596 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.596 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.596 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.596 DEBUG:boto:Data: 2024-03-04T16:33:40.597 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.597 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.597 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.597 DEBUG:boto:Params: {} 2024-03-04T16:33:40.597 DEBUG:boto:Token: None 2024-03-04T16:33:40.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+et05rTYSWEEQirdhPXyGi1C8Fs= 2024-03-04T16:33:40.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+et05rTYSWEEQirdhPXyGi1C8Fs=', 'Content-Length': '0'} 2024-03-04T16:33:40.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f0b7f467b74916e-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.599 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.599 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.599 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.600 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.600 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.600 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.600 DEBUG:boto:Data: 2024-03-04T16:33:40.600 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.600 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.600 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.600 DEBUG:boto:Params: {} 2024-03-04T16:33:40.600 DEBUG:boto:Token: None 2024-03-04T16:33:40.600 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+et05rTYSWEEQirdhPXyGi1C8Fs= 2024-03-04T16:33:40.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+et05rTYSWEEQirdhPXyGi1C8Fs=', 'Content-Length': '0'} 2024-03-04T16:33:40.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c575b4b64f82b9a3-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.603 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.603 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.603 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.603 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.603 DEBUG:boto:Method: GET 2024-03-04T16:33:40.603 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.603 DEBUG:boto:Data: 2024-03-04T16:33:40.603 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.603 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.603 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.603 DEBUG:boto:Params: {} 2024-03-04T16:33:40.604 DEBUG:boto:Token: None 2024-03-04T16:33:40.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.604 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo= 2024-03-04T16:33:40.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo=', 'Content-Length': '0'} 2024-03-04T16:33:40.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc3cbd8dfe914351-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.606 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.606 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.606 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.606 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.606 DEBUG:boto:Method: GET 2024-03-04T16:33:40.607 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.607 DEBUG:boto:Data: 2024-03-04T16:33:40.607 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.607 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.607 DEBUG:boto:Params: {} 2024-03-04T16:33:40.607 DEBUG:boto:Token: None 2024-03-04T16:33:40.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:40.607 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo= 2024-03-04T16:33:40.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HJGL6WtxN33LGI6tL3Iwcr4OZKo=', 'Content-Length': '0'} 2024-03-04T16:33:40.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e930208a889408dc-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.609 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.609 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.609 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.610 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.610 DEBUG:boto:Method: GET 2024-03-04T16:33:40.610 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.610 DEBUG:boto:Data: 2024-03-04T16:33:40.610 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.610 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.610 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.610 DEBUG:boto:Params: {} 2024-03-04T16:33:40.610 DEBUG:boto:Token: None 2024-03-04T16:33:40.610 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.611 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955c8e4a09936423-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.613 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.614 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.614 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.614 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.614 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.614 DEBUG:boto:Method: GET 2024-03-04T16:33:40.614 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.614 DEBUG:boto:Data: 2024-03-04T16:33:40.614 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.614 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.614 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.615 DEBUG:boto:Params: {} 2024-03-04T16:33:40.615 DEBUG:boto:Token: None 2024-03-04T16:33:40.615 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.615 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007137dc42cfddb0be-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.618 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.618 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.618 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.618 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.618 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.618 DEBUG:boto:Data: 2024-03-04T16:33:40.619 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.619 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.619 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.619 DEBUG:boto:Params: {} 2024-03-04T16:33:40.619 DEBUG:boto:Token: None 2024-03-04T16:33:40.619 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.619 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000070238a529a11788-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.624 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.624 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.625 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.625 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.625 DEBUG:boto:Data: 2024-03-04T16:33:40.625 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.625 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.625 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.625 DEBUG:boto:Params: {} 2024-03-04T16:33:40.625 DEBUG:boto:Token: None 2024-03-04T16:33:40.625 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061fcab8f1ed57116-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.629 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.629 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.629 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.629 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.629 DEBUG:boto:Method: GET 2024-03-04T16:33:40.629 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.630 DEBUG:boto:Data: 2024-03-04T16:33:40.630 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.630 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.630 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.630 DEBUG:boto:Params: {} 2024-03-04T16:33:40.630 DEBUG:boto:Token: None 2024-03-04T16:33:40.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce3ed2ab43f9ca78-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.680 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.680 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.680 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.680 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.681 DEBUG:boto:Method: GET 2024-03-04T16:33:40.681 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.681 DEBUG:boto:Data: 2024-03-04T16:33:40.681 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.681 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.681 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.681 DEBUG:boto:Params: {} 2024-03-04T16:33:40.681 DEBUG:boto:Token: None 2024-03-04T16:33:40.681 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.682 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b1a4261e5910c6e-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.684 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.684 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.684 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.684 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.684 DEBUG:boto:Method: GET 2024-03-04T16:33:40.685 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.685 DEBUG:boto:Data: 2024-03-04T16:33:40.685 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.685 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.685 DEBUG:boto:Params: {} 2024-03-04T16:33:40.685 DEBUG:boto:Token: None 2024-03-04T16:33:40.685 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.685 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a435f61f17312d-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.688 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.689 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.689 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.689 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.689 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.689 DEBUG:boto:Method: GET 2024-03-04T16:33:40.689 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.689 DEBUG:boto:Data: 2024-03-04T16:33:40.689 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.690 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.690 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.690 DEBUG:boto:Params: {} 2024-03-04T16:33:40.690 DEBUG:boto:Token: None 2024-03-04T16:33:40.690 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009229e0602dee5d37-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.692 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.693 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.693 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.693 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.693 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.693 DEBUG:boto:Method: GET 2024-03-04T16:33:40.693 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.693 DEBUG:boto:Data: 2024-03-04T16:33:40.694 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.694 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.694 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.694 DEBUG:boto:Params: {} 2024-03-04T16:33:40.694 DEBUG:boto:Token: None 2024-03-04T16:33:40.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU= 2024-03-04T16:33:40.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU=', 'Content-Length': '0'} 2024-03-04T16:33:40.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000689283eab3dcd4ca-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.744 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.744 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.744 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.744 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.745 DEBUG:boto:Method: GET 2024-03-04T16:33:40.745 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.745 DEBUG:boto:Data: 2024-03-04T16:33:40.745 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.745 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.745 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.745 DEBUG:boto:Params: {} 2024-03-04T16:33:40.745 DEBUG:boto:Token: None 2024-03-04T16:33:40.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU= 2024-03-04T16:33:40.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU=', 'Content-Length': '0'} 2024-03-04T16:33:40.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006f7d89c577cb9e3-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.748 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.748 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.748 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.748 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.748 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.748 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.748 DEBUG:boto:Data: 2024-03-04T16:33:40.748 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.749 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.749 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.749 DEBUG:boto:Params: {} 2024-03-04T16:33:40.749 DEBUG:boto:Token: None 2024-03-04T16:33:40.749 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.749 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0OmQJLoGQ+LdeVtFWFYaO/1fkB0= 2024-03-04T16:33:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0OmQJLoGQ+LdeVtFWFYaO/1fkB0=', 'Content-Length': '0'} 2024-03-04T16:33:40.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f356fc1d5a2bc4c4-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.751 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.751 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.751 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.751 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.752 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.752 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.752 DEBUG:boto:Data: 2024-03-04T16:33:40.752 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.752 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.752 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.752 DEBUG:boto:Params: {} 2024-03-04T16:33:40.752 DEBUG:boto:Token: None 2024-03-04T16:33:40.752 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.752 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0OmQJLoGQ+LdeVtFWFYaO/1fkB0= 2024-03-04T16:33:40.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0OmQJLoGQ+LdeVtFWFYaO/1fkB0=', 'Content-Length': '0'} 2024-03-04T16:33:40.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b757c68ee9ffec1-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.755 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.755 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.755 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.755 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.755 DEBUG:boto:Method: GET 2024-03-04T16:33:40.755 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.755 DEBUG:boto:Data: 2024-03-04T16:33:40.755 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.755 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.755 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.755 DEBUG:boto:Params: {} 2024-03-04T16:33:40.756 DEBUG:boto:Token: None 2024-03-04T16:33:40.756 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.756 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU= 2024-03-04T16:33:40.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU=', 'Content-Length': '0'} 2024-03-04T16:33:40.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000153dfe625d184769-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.804 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.804 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.804 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.805 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.805 DEBUG:boto:Method: GET 2024-03-04T16:33:40.805 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.805 DEBUG:boto:Data: 2024-03-04T16:33:40.805 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.805 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.805 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.805 DEBUG:boto:Params: {} 2024-03-04T16:33:40.805 DEBUG:boto:Token: None 2024-03-04T16:33:40.805 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:40.806 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU= 2024-03-04T16:33:40.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9jtOs90Pwz14YEK2Lga9vo8CzSU=', 'Content-Length': '0'} 2024-03-04T16:33:40.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008ac60573720a8b1-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.808 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.808 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.808 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.808 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.808 DEBUG:boto:Method: GET 2024-03-04T16:33:40.808 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.808 DEBUG:boto:Data: 2024-03-04T16:33:40.808 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.808 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.809 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.809 DEBUG:boto:Params: {} 2024-03-04T16:33:40.809 DEBUG:boto:Token: None 2024-03-04T16:33:40.809 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.809 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f750677da249ec9-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.812 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.812 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.812 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.813 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.813 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.813 DEBUG:boto:Method: GET 2024-03-04T16:33:40.813 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.813 DEBUG:boto:Data: 2024-03-04T16:33:40.813 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.813 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.813 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.813 DEBUG:boto:Params: {} 2024-03-04T16:33:40.813 DEBUG:boto:Token: None 2024-03-04T16:33:40.813 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe1e39eaec84671-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.816 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.816 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.817 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.817 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.817 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.817 DEBUG:boto:Data: 2024-03-04T16:33:40.817 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.817 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.817 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.817 DEBUG:boto:Params: {} 2024-03-04T16:33:40.817 DEBUG:boto:Token: None 2024-03-04T16:33:40.817 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.817 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d95d5b3e9ab10ca9-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.824 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.824 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.824 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.824 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.824 DEBUG:boto:Data: 2024-03-04T16:33:40.825 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.825 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.825 DEBUG:boto:Params: {} 2024-03-04T16:33:40.825 DEBUG:boto:Token: None 2024-03-04T16:33:40.825 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.825 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070fd03d0a21887e8-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.829 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.829 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.829 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.829 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.829 DEBUG:boto:Method: GET 2024-03-04T16:33:40.829 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.829 DEBUG:boto:Data: 2024-03-04T16:33:40.829 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.829 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.829 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.829 DEBUG:boto:Params: {} 2024-03-04T16:33:40.830 DEBUG:boto:Token: None 2024-03-04T16:33:40.830 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002848989e4b15a449-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.872 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.872 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.872 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.873 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.873 DEBUG:boto:Method: GET 2024-03-04T16:33:40.873 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.873 DEBUG:boto:Data: 2024-03-04T16:33:40.873 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.873 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.873 DEBUG:boto:Params: {} 2024-03-04T16:33:40.874 DEBUG:boto:Token: None 2024-03-04T16:33:40.874 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c9eee8579ff91af-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.876 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.876 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.876 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.876 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.876 DEBUG:boto:Method: GET 2024-03-04T16:33:40.876 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.876 DEBUG:boto:Data: 2024-03-04T16:33:40.876 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.877 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.877 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.877 DEBUG:boto:Params: {} 2024-03-04T16:33:40.877 DEBUG:boto:Token: None 2024-03-04T16:33:40.877 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.877 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004bc1f6fa2776b3c-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.880 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.881 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.881 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.881 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.881 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.881 DEBUG:boto:Method: GET 2024-03-04T16:33:40.881 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.881 DEBUG:boto:Data: 2024-03-04T16:33:40.881 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.881 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.881 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.881 DEBUG:boto:Params: {} 2024-03-04T16:33:40.882 DEBUG:boto:Token: None 2024-03-04T16:33:40.882 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.882 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9971f39baad2472-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.885 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.885 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.885 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.885 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.885 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.885 DEBUG:boto:Method: GET 2024-03-04T16:33:40.885 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.886 DEBUG:boto:Data: 2024-03-04T16:33:40.886 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.886 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.886 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.886 DEBUG:boto:Params: {} 2024-03-04T16:33:40.886 DEBUG:boto:Token: None 2024-03-04T16:33:40.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8= 2024-03-04T16:33:40.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8=', 'Content-Length': '0'} 2024-03-04T16:33:40.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006febaba0f2f4ba95-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.888 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.889 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.889 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.889 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.889 DEBUG:boto:Method: GET 2024-03-04T16:33:40.889 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.889 DEBUG:boto:Data: 2024-03-04T16:33:40.889 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.889 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.889 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.889 DEBUG:boto:Params: {} 2024-03-04T16:33:40.889 DEBUG:boto:Token: None 2024-03-04T16:33:40.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8= 2024-03-04T16:33:40.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8=', 'Content-Length': '0'} 2024-03-04T16:33:40.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c28150bc65caf01-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.892 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.892 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.892 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.892 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.892 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.892 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.893 DEBUG:boto:Data: 2024-03-04T16:33:40.893 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.893 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.893 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.893 DEBUG:boto:Params: {} 2024-03-04T16:33:40.893 DEBUG:boto:Token: None 2024-03-04T16:33:40.893 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.893 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vsJ8BFqajhJnXjUsvJDvxOXnHGk= 2024-03-04T16:33:40.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vsJ8BFqajhJnXjUsvJDvxOXnHGk=', 'Content-Length': '0'} 2024-03-04T16:33:40.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cda41314673464d9-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.895 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.895 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.895 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.896 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.896 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.896 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.896 DEBUG:boto:Data: 2024-03-04T16:33:40.896 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.896 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.896 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.897 DEBUG:boto:Params: {} 2024-03-04T16:33:40.897 DEBUG:boto:Token: None 2024-03-04T16:33:40.897 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.897 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vsJ8BFqajhJnXjUsvJDvxOXnHGk= 2024-03-04T16:33:40.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vsJ8BFqajhJnXjUsvJDvxOXnHGk=', 'Content-Length': '0'} 2024-03-04T16:33:40.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000096ebffc6641ffd0-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.899 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.899 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.899 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.899 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.899 DEBUG:boto:Method: GET 2024-03-04T16:33:40.900 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.900 DEBUG:boto:Data: 2024-03-04T16:33:40.900 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.900 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.900 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.900 DEBUG:boto:Params: {} 2024-03-04T16:33:40.900 DEBUG:boto:Token: None 2024-03-04T16:33:40.900 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.900 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8= 2024-03-04T16:33:40.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8=', 'Content-Length': '0'} 2024-03-04T16:33:40.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e716d72afcb8f29a-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.944 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.944 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.944 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.945 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.945 DEBUG:boto:Method: GET 2024-03-04T16:33:40.945 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.945 DEBUG:boto:Data: 2024-03-04T16:33:40.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.945 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.945 DEBUG:boto:Params: {} 2024-03-04T16:33:40.945 DEBUG:boto:Token: None 2024-03-04T16:33:40.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:40.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8= 2024-03-04T16:33:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSzDrSRt7Uc5+XQ+lu4NV0co9c8=', 'Content-Length': '0'} 2024-03-04T16:33:40.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb8626025a3d11c3-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.948 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.948 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.948 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.948 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.948 DEBUG:boto:Method: GET 2024-03-04T16:33:40.948 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.948 DEBUG:boto:Data: 2024-03-04T16:33:40.948 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.948 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.948 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.949 DEBUG:boto:Params: {} 2024-03-04T16:33:40.949 DEBUG:boto:Token: None 2024-03-04T16:33:40.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b2f50ba91fd22f1-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.952 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.953 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:40.953 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:40.953 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.953 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:40.953 DEBUG:boto:Method: GET 2024-03-04T16:33:40.953 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:40.953 DEBUG:boto:Data: 2024-03-04T16:33:40.953 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.953 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.953 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.954 DEBUG:boto:Params: {} 2024-03-04T16:33:40.954 DEBUG:boto:Token: None 2024-03-04T16:33:40.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/ 2024-03-04T16:33:40.954 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU= 2024-03-04T16:33:40.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wJy7TxT2ISMB90PH7ndgDH2lrhU=', 'Content-Length': '0'} 2024-03-04T16:33:40.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786b97f15ddc4412-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.957 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:40.958 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.958 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.958 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.958 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.958 DEBUG:boto:Data: 2024-03-04T16:33:40.958 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.958 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.958 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.958 DEBUG:boto:Params: {} 2024-03-04T16:33:40.958 DEBUG:boto:Token: None 2024-03-04T16:33:40.959 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ac981c4fba907b9-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.962 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.963 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.963 DEBUG:boto:Method: HEAD 2024-03-04T16:33:40.963 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:40.963 DEBUG:boto:Data: 2024-03-04T16:33:40.963 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.963 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:40.963 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.963 DEBUG:boto:Params: {} 2024-03-04T16:33:40.963 DEBUG:boto:Token: None 2024-03-04T16:33:40.964 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj 2024-03-04T16:33:40.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8= 2024-03-04T16:33:40.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LAOcw3ws/DkJnfKL1QMkfmTQ6e8=', 'Content-Length': '0'} 2024-03-04T16:33:40.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2a605bcf8f013a8-0065e5f7e4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:40.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:40.967 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:40.967 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:40.967 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.967 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.968 DEBUG:boto:Method: GET 2024-03-04T16:33:40.968 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.968 DEBUG:boto:Data: 2024-03-04T16:33:40.968 DEBUG:boto:Headers: {} 2024-03-04T16:33:40.968 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:40.968 DEBUG:boto:Port: 8000 2024-03-04T16:33:40.968 DEBUG:boto:Params: {} 2024-03-04T16:33:40.968 DEBUG:boto:Token: None 2024-03-04T16:33:40.968 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:40 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:40.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY= 2024-03-04T16:33:40.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ipQeSCNr21qMUquCm78kP1vkxsY=', 'Content-Length': '0'} 2024-03-04T16:33:40.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5964c2da670b939-0065e5f7e4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.012 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.012 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.012 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.013 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.013 DEBUG:boto:Method: GET 2024-03-04T16:33:41.013 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.013 DEBUG:boto:Data: 2024-03-04T16:33:41.013 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.013 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.013 DEBUG:boto:Params: {} 2024-03-04T16:33:41.013 DEBUG:boto:Token: None 2024-03-04T16:33:41.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c4e6b3a8f8af136-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.056 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.056 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.056 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.057 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.057 DEBUG:boto:Method: GET 2024-03-04T16:33:41.057 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.057 DEBUG:boto:Data: 2024-03-04T16:33:41.057 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.057 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.057 DEBUG:boto:Params: {} 2024-03-04T16:33:41.057 DEBUG:boto:Token: None 2024-03-04T16:33:41.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ddf551b6b1e19ed-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.060 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.061 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.061 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.061 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.061 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.061 DEBUG:boto:Method: GET 2024-03-04T16:33:41.061 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.061 DEBUG:boto:Data: 2024-03-04T16:33:41.062 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.062 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.062 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.062 DEBUG:boto:Params: {} 2024-03-04T16:33:41.062 DEBUG:boto:Token: None 2024-03-04T16:33:41.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7da137d3d05a48d-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.108 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.109 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.109 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.109 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.109 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.109 DEBUG:boto:Method: GET 2024-03-04T16:33:41.109 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.109 DEBUG:boto:Data: 2024-03-04T16:33:41.109 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.109 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.110 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.110 DEBUG:boto:Params: {} 2024-03-04T16:33:41.110 DEBUG:boto:Token: None 2024-03-04T16:33:41.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco= 2024-03-04T16:33:41.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco=', 'Content-Length': '0'} 2024-03-04T16:33:41.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f867d85e71c0b7c0-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.156 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.156 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.156 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.157 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.157 DEBUG:boto:Method: GET 2024-03-04T16:33:41.157 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.157 DEBUG:boto:Data: 2024-03-04T16:33:41.157 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.157 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.157 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.157 DEBUG:boto:Params: {} 2024-03-04T16:33:41.157 DEBUG:boto:Token: None 2024-03-04T16:33:41.158 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.158 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco= 2024-03-04T16:33:41.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco=', 'Content-Length': '0'} 2024-03-04T16:33:41.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021db8a7b21539e94-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.160 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.160 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.161 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.161 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.161 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.161 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.161 DEBUG:boto:Data: 2024-03-04T16:33:41.161 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.161 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.161 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.161 DEBUG:boto:Params: {} 2024-03-04T16:33:41.161 DEBUG:boto:Token: None 2024-03-04T16:33:41.161 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.162 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9OT4TyfpEzNmKvnZqBCqxdUPBMI= 2024-03-04T16:33:41.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9OT4TyfpEzNmKvnZqBCqxdUPBMI=', 'Content-Length': '0'} 2024-03-04T16:33:41.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1b07abc4bd50581-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.164 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.164 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.164 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.164 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.164 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.164 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.164 DEBUG:boto:Data: 2024-03-04T16:33:41.164 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.164 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.165 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.165 DEBUG:boto:Params: {} 2024-03-04T16:33:41.165 DEBUG:boto:Token: None 2024-03-04T16:33:41.165 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9OT4TyfpEzNmKvnZqBCqxdUPBMI= 2024-03-04T16:33:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9OT4TyfpEzNmKvnZqBCqxdUPBMI=', 'Content-Length': '0'} 2024-03-04T16:33:41.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006526ea242711ca67-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.167 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.167 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.167 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.168 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.168 DEBUG:boto:Method: GET 2024-03-04T16:33:41.168 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.168 DEBUG:boto:Data: 2024-03-04T16:33:41.168 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.168 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.168 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.168 DEBUG:boto:Params: {} 2024-03-04T16:33:41.168 DEBUG:boto:Token: None 2024-03-04T16:33:41.168 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco= 2024-03-04T16:33:41.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco=', 'Content-Length': '0'} 2024-03-04T16:33:41.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000429cf36691ed1a0d-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.212 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.212 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.212 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.213 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.213 DEBUG:boto:Method: GET 2024-03-04T16:33:41.213 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.213 DEBUG:boto:Data: 2024-03-04T16:33:41.213 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.213 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.213 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.213 DEBUG:boto:Params: {} 2024-03-04T16:33:41.213 DEBUG:boto:Token: None 2024-03-04T16:33:41.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:41.214 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco= 2024-03-04T16:33:41.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L8R+CBH32Hkcm+Gti7XL+eHQtco=', 'Content-Length': '0'} 2024-03-04T16:33:41.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064a82fb7a90a6108-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.216 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.216 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.216 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.217 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.217 DEBUG:boto:Method: GET 2024-03-04T16:33:41.217 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.217 DEBUG:boto:Data: 2024-03-04T16:33:41.217 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.217 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.217 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.217 DEBUG:boto:Params: {} 2024-03-04T16:33:41.217 DEBUG:boto:Token: None 2024-03-04T16:33:41.217 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.218 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a94d88439d9de7-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.220 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.221 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.221 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.221 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.221 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.221 DEBUG:boto:Method: GET 2024-03-04T16:33:41.221 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.221 DEBUG:boto:Data: 2024-03-04T16:33:41.221 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.222 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.222 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.222 DEBUG:boto:Params: {} 2024-03-04T16:33:41.222 DEBUG:boto:Token: None 2024-03-04T16:33:41.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2e36fc977f6279c-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.225 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.225 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.225 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.226 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.226 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.226 DEBUG:boto:Data: 2024-03-04T16:33:41.226 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.226 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.226 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.226 DEBUG:boto:Params: {} 2024-03-04T16:33:41.226 DEBUG:boto:Token: None 2024-03-04T16:33:41.226 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081c17f2506010055-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.230 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.230 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.230 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.230 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.230 DEBUG:boto:Data: 2024-03-04T16:33:41.230 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.230 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.230 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.231 DEBUG:boto:Params: {} 2024-03-04T16:33:41.231 DEBUG:boto:Token: None 2024-03-04T16:33:41.231 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.231 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059cc493c7e8883b1-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.236 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.236 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.236 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.236 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.236 DEBUG:boto:Method: GET 2024-03-04T16:33:41.237 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.237 DEBUG:boto:Data: 2024-03-04T16:33:41.237 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.237 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.237 DEBUG:boto:Params: {} 2024-03-04T16:33:41.237 DEBUG:boto:Token: None 2024-03-04T16:33:41.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.237 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002325ca267f5ec9fa-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.284 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.284 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.284 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.284 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.285 DEBUG:boto:Method: GET 2024-03-04T16:33:41.285 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.285 DEBUG:boto:Data: 2024-03-04T16:33:41.285 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.285 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.285 DEBUG:boto:Params: {} 2024-03-04T16:33:41.285 DEBUG:boto:Token: None 2024-03-04T16:33:41.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.285 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2d159dfd18565f4-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.332 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.332 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.332 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.333 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.333 DEBUG:boto:Method: GET 2024-03-04T16:33:41.333 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.333 DEBUG:boto:Data: 2024-03-04T16:33:41.333 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.333 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.333 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.333 DEBUG:boto:Params: {} 2024-03-04T16:33:41.333 DEBUG:boto:Token: None 2024-03-04T16:33:41.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.334 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000134a283706041a55-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.337 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.337 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.337 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.337 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.338 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.338 DEBUG:boto:Method: GET 2024-03-04T16:33:41.338 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.338 DEBUG:boto:Data: 2024-03-04T16:33:41.338 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.338 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.338 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.338 DEBUG:boto:Params: {} 2024-03-04T16:33:41.338 DEBUG:boto:Token: None 2024-03-04T16:33:41.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74c4f6108b05e3e-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.342 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.342 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.343 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.343 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.343 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.343 DEBUG:boto:Method: GET 2024-03-04T16:33:41.343 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.343 DEBUG:boto:Data: 2024-03-04T16:33:41.343 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.343 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.343 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.343 DEBUG:boto:Params: {} 2024-03-04T16:33:41.343 DEBUG:boto:Token: None 2024-03-04T16:33:41.344 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.344 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE= 2024-03-04T16:33:41.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE=', 'Content-Length': '0'} 2024-03-04T16:33:41.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000420e2d2bd9379b6c-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.392 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.392 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.392 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.392 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.393 DEBUG:boto:Method: GET 2024-03-04T16:33:41.393 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.393 DEBUG:boto:Data: 2024-03-04T16:33:41.393 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.393 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.393 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.393 DEBUG:boto:Params: {} 2024-03-04T16:33:41.393 DEBUG:boto:Token: None 2024-03-04T16:33:41.393 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE= 2024-03-04T16:33:41.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE=', 'Content-Length': '0'} 2024-03-04T16:33:41.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a9419eed8151896-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.397 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.397 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.397 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.397 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.397 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.397 DEBUG:boto:Data: 2024-03-04T16:33:41.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.398 DEBUG:boto:Params: {} 2024-03-04T16:33:41.398 DEBUG:boto:Token: None 2024-03-04T16:33:41.398 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zCueXcuPpuuBuFYcfl9hmGMzTnk= 2024-03-04T16:33:41.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zCueXcuPpuuBuFYcfl9hmGMzTnk=', 'Content-Length': '0'} 2024-03-04T16:33:41.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdd095b51aeafd73-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.401 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.401 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.401 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.401 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.401 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.401 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.401 DEBUG:boto:Data: 2024-03-04T16:33:41.401 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.401 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.401 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.401 DEBUG:boto:Params: {} 2024-03-04T16:33:41.401 DEBUG:boto:Token: None 2024-03-04T16:33:41.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zCueXcuPpuuBuFYcfl9hmGMzTnk= 2024-03-04T16:33:41.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zCueXcuPpuuBuFYcfl9hmGMzTnk=', 'Content-Length': '0'} 2024-03-04T16:33:41.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f30a858ea366521f-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.404 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.404 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.404 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.404 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.405 DEBUG:boto:Method: GET 2024-03-04T16:33:41.405 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.405 DEBUG:boto:Data: 2024-03-04T16:33:41.405 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.405 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.405 DEBUG:boto:Params: {} 2024-03-04T16:33:41.405 DEBUG:boto:Token: None 2024-03-04T16:33:41.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE= 2024-03-04T16:33:41.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE=', 'Content-Length': '0'} 2024-03-04T16:33:41.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017f7df05c4c6e3f8-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.452 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.453 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.453 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.453 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.453 DEBUG:boto:Method: GET 2024-03-04T16:33:41.453 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.453 DEBUG:boto:Data: 2024-03-04T16:33:41.453 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.453 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.453 DEBUG:boto:Params: {} 2024-03-04T16:33:41.454 DEBUG:boto:Token: None 2024-03-04T16:33:41.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:41.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE= 2024-03-04T16:33:41.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bpEkJN7hJ5J1vzCHipTpXvHMYzE=', 'Content-Length': '0'} 2024-03-04T16:33:41.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063e57a20cdd59d9a-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.457 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.457 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.457 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.457 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.457 DEBUG:boto:Method: GET 2024-03-04T16:33:41.457 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.458 DEBUG:boto:Data: 2024-03-04T16:33:41.458 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.458 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.458 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.458 DEBUG:boto:Params: {} 2024-03-04T16:33:41.458 DEBUG:boto:Token: None 2024-03-04T16:33:41.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f001162beeacb509-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.461 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.462 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.462 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.462 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.462 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.462 DEBUG:boto:Method: GET 2024-03-04T16:33:41.462 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.462 DEBUG:boto:Data: 2024-03-04T16:33:41.462 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.462 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.463 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.463 DEBUG:boto:Params: {} 2024-03-04T16:33:41.463 DEBUG:boto:Token: None 2024-03-04T16:33:41.463 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.463 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d083c421b37d227-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.512 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.513 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.513 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.513 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.513 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.513 DEBUG:boto:Data: 2024-03-04T16:33:41.513 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.513 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.513 DEBUG:boto:Params: {} 2024-03-04T16:33:41.514 DEBUG:boto:Token: None 2024-03-04T16:33:41.514 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070bef61fcf274124-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.518 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.518 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.518 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.518 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.518 DEBUG:boto:Data: 2024-03-04T16:33:41.518 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.518 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.518 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.518 DEBUG:boto:Params: {} 2024-03-04T16:33:41.518 DEBUG:boto:Token: None 2024-03-04T16:33:41.519 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b9e4eb47e0b88d7-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.523 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.523 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.523 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.523 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.523 DEBUG:boto:Method: GET 2024-03-04T16:33:41.523 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.523 DEBUG:boto:Data: 2024-03-04T16:33:41.524 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.524 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.524 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.524 DEBUG:boto:Params: {} 2024-03-04T16:33:41.524 DEBUG:boto:Token: None 2024-03-04T16:33:41.524 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009307330baf36062d-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.527 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.527 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.527 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.527 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.527 DEBUG:boto:Method: GET 2024-03-04T16:33:41.528 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.528 DEBUG:boto:Data: 2024-03-04T16:33:41.528 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.528 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.528 DEBUG:boto:Params: {} 2024-03-04T16:33:41.528 DEBUG:boto:Token: None 2024-03-04T16:33:41.528 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087a795fe7834edc1-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.576 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.576 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.576 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.577 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.577 DEBUG:boto:Method: GET 2024-03-04T16:33:41.577 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.577 DEBUG:boto:Data: 2024-03-04T16:33:41.577 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.577 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.577 DEBUG:boto:Params: {} 2024-03-04T16:33:41.577 DEBUG:boto:Token: None 2024-03-04T16:33:41.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc9b4ab5fc5c011-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.580 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.581 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.581 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.581 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.581 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.581 DEBUG:boto:Method: GET 2024-03-04T16:33:41.581 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.581 DEBUG:boto:Data: 2024-03-04T16:33:41.581 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.582 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.582 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.582 DEBUG:boto:Params: {} 2024-03-04T16:33:41.582 DEBUG:boto:Token: None 2024-03-04T16:33:41.582 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.582 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd8af7f2a71078fa-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.628 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.629 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.629 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.629 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.629 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.629 DEBUG:boto:Method: GET 2024-03-04T16:33:41.629 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.629 DEBUG:boto:Data: 2024-03-04T16:33:41.629 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.630 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.630 DEBUG:boto:Params: {} 2024-03-04T16:33:41.630 DEBUG:boto:Token: None 2024-03-04T16:33:41.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY= 2024-03-04T16:33:41.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY=', 'Content-Length': '0'} 2024-03-04T16:33:41.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a882b10f967db6e-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.632 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.632 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.633 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.633 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.633 DEBUG:boto:Method: GET 2024-03-04T16:33:41.633 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.633 DEBUG:boto:Data: 2024-03-04T16:33:41.633 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.633 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.633 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.633 DEBUG:boto:Params: {} 2024-03-04T16:33:41.633 DEBUG:boto:Token: None 2024-03-04T16:33:41.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY= 2024-03-04T16:33:41.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY=', 'Content-Length': '0'} 2024-03-04T16:33:41.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7fa305e551572c8-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.636 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.636 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.636 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.636 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.637 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.637 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.637 DEBUG:boto:Data: 2024-03-04T16:33:41.637 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.637 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.637 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.637 DEBUG:boto:Params: {} 2024-03-04T16:33:41.637 DEBUG:boto:Token: None 2024-03-04T16:33:41.637 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.637 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LB6kggzWcfd0QPiVMB0RZ5kPFpE= 2024-03-04T16:33:41.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LB6kggzWcfd0QPiVMB0RZ5kPFpE=', 'Content-Length': '0'} 2024-03-04T16:33:41.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027bf1a49c7b1e1cf-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.640 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.640 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.640 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.640 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.640 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.640 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.640 DEBUG:boto:Data: 2024-03-04T16:33:41.640 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.641 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.641 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.641 DEBUG:boto:Params: {} 2024-03-04T16:33:41.644 DEBUG:boto:Token: None 2024-03-04T16:33:41.644 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.644 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LB6kggzWcfd0QPiVMB0RZ5kPFpE= 2024-03-04T16:33:41.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LB6kggzWcfd0QPiVMB0RZ5kPFpE=', 'Content-Length': '0'} 2024-03-04T16:33:41.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000939df44affad61a3-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.646 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.646 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.647 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.647 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.647 DEBUG:boto:Method: GET 2024-03-04T16:33:41.647 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.647 DEBUG:boto:Data: 2024-03-04T16:33:41.647 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.647 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.647 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.647 DEBUG:boto:Params: {} 2024-03-04T16:33:41.647 DEBUG:boto:Token: None 2024-03-04T16:33:41.647 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.648 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY= 2024-03-04T16:33:41.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY=', 'Content-Length': '0'} 2024-03-04T16:33:41.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003169ebc57c44ed92-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.696 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.696 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.696 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.696 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.697 DEBUG:boto:Method: GET 2024-03-04T16:33:41.697 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.697 DEBUG:boto:Data: 2024-03-04T16:33:41.697 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.697 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.697 DEBUG:boto:Params: {} 2024-03-04T16:33:41.697 DEBUG:boto:Token: None 2024-03-04T16:33:41.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:41.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY= 2024-03-04T16:33:41.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9IsehAFmHrsxZnZPlownvQK0tfY=', 'Content-Length': '0'} 2024-03-04T16:33:41.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd69bc6c9bffb21c-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.740 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.740 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.740 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.741 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.741 DEBUG:boto:Method: GET 2024-03-04T16:33:41.741 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.741 DEBUG:boto:Data: 2024-03-04T16:33:41.741 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.741 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.741 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.741 DEBUG:boto:Params: {} 2024-03-04T16:33:41.741 DEBUG:boto:Token: None 2024-03-04T16:33:41.741 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.742 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210b4886d3606f91-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.745 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.745 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.745 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.745 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.745 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.745 DEBUG:boto:Method: GET 2024-03-04T16:33:41.746 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.746 DEBUG:boto:Data: 2024-03-04T16:33:41.746 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.746 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.746 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.746 DEBUG:boto:Params: {} 2024-03-04T16:33:41.746 DEBUG:boto:Token: None 2024-03-04T16:33:41.746 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.746 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1476f6e0ce3ba8-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.792 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.793 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.793 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.793 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.793 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.793 DEBUG:boto:Data: 2024-03-04T16:33:41.793 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.793 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.793 DEBUG:boto:Params: {} 2024-03-04T16:33:41.794 DEBUG:boto:Token: None 2024-03-04T16:33:41.794 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009775a012f8ec6564-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.797 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.797 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.798 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.798 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:41.798 DEBUG:boto:Data: 2024-03-04T16:33:41.798 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.798 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.798 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.798 DEBUG:boto:Params: {} 2024-03-04T16:33:41.798 DEBUG:boto:Token: None 2024-03-04T16:33:41.798 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj 2024-03-04T16:33:41.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0= 2024-03-04T16:33:41.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I8edD1fzU55E4VEz2ql22MdRVs0=', 'Content-Length': '0'} 2024-03-04T16:33:41.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f23776f26649a9-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.802 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.802 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.802 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.803 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.803 DEBUG:boto:Method: GET 2024-03-04T16:33:41.803 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.803 DEBUG:boto:Data: 2024-03-04T16:33:41.803 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.803 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.803 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.803 DEBUG:boto:Params: {} 2024-03-04T16:33:41.803 DEBUG:boto:Token: None 2024-03-04T16:33:41.803 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.804 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008de1feb928aea98d-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.806 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.806 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.806 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.806 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.806 DEBUG:boto:Method: GET 2024-03-04T16:33:41.806 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.806 DEBUG:boto:Data: 2024-03-04T16:33:41.806 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.806 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.806 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.807 DEBUG:boto:Params: {} 2024-03-04T16:33:41.807 DEBUG:boto:Token: None 2024-03-04T16:33:41.807 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:41.807 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0= 2024-03-04T16:33:41.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjDnN3hWFwtgZ+R8TTmqnTV7Gj0=', 'Content-Length': '0'} 2024-03-04T16:33:41.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0ada19f4be7c6f6-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.852 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.852 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.852 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.853 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.853 DEBUG:boto:Method: GET 2024-03-04T16:33:41.853 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.853 DEBUG:boto:Data: 2024-03-04T16:33:41.853 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.853 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.853 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.853 DEBUG:boto:Params: {} 2024-03-04T16:33:41.853 DEBUG:boto:Token: None 2024-03-04T16:33:41.854 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.854 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000591ef128185a8f92-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.857 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.857 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:41.857 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:41.857 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.857 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:41.857 DEBUG:boto:Method: GET 2024-03-04T16:33:41.858 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:41.858 DEBUG:boto:Data: 2024-03-04T16:33:41.858 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.858 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.858 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.858 DEBUG:boto:Params: {} 2024-03-04T16:33:41.858 DEBUG:boto:Token: None 2024-03-04T16:33:41.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/ 2024-03-04T16:33:41.858 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI= 2024-03-04T16:33:41.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D6sKPEpcsg5QWvgbWE0dm0/qaaI=', 'Content-Length': '0'} 2024-03-04T16:33:41.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000540d7e233373fd93-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.908 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:41.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.909 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.909 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.909 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.909 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.909 DEBUG:boto:Method: GET 2024-03-04T16:33:41.909 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.909 DEBUG:boto:Data: 2024-03-04T16:33:41.909 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.910 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.910 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.910 DEBUG:boto:Params: {} 2024-03-04T16:33:41.910 DEBUG:boto:Token: None 2024-03-04T16:33:41.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M= 2024-03-04T16:33:41.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M=', 'Content-Length': '0'} 2024-03-04T16:33:41.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e72ef1dfd96cfbc-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.956 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.956 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.956 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.957 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.957 DEBUG:boto:Method: GET 2024-03-04T16:33:41.957 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.957 DEBUG:boto:Data: 2024-03-04T16:33:41.957 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.957 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.957 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.957 DEBUG:boto:Params: {} 2024-03-04T16:33:41.957 DEBUG:boto:Token: None 2024-03-04T16:33:41.957 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.958 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M= 2024-03-04T16:33:41.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M=', 'Content-Length': '0'} 2024-03-04T16:33:41.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4a2af2356a83f2d-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.960 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.960 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.960 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.960 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.961 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.961 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.961 DEBUG:boto:Data: 2024-03-04T16:33:41.961 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.961 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.961 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.961 DEBUG:boto:Params: {} 2024-03-04T16:33:41.961 DEBUG:boto:Token: None 2024-03-04T16:33:41.961 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.961 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M5J7IxEf4/i6CQlXMlOBKwJyRJ8= 2024-03-04T16:33:41.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M5J7IxEf4/i6CQlXMlOBKwJyRJ8=', 'Content-Length': '0'} 2024-03-04T16:33:41.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000453df012b5c3f60-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.964 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.964 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.964 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.964 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.964 DEBUG:boto:Method: HEAD 2024-03-04T16:33:41.964 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.964 DEBUG:boto:Data: 2024-03-04T16:33:41.964 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.964 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:41.964 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.964 DEBUG:boto:Params: {} 2024-03-04T16:33:41.965 DEBUG:boto:Token: None 2024-03-04T16:33:41.965 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.965 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M5J7IxEf4/i6CQlXMlOBKwJyRJ8= 2024-03-04T16:33:41.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M5J7IxEf4/i6CQlXMlOBKwJyRJ8=', 'Content-Length': '0'} 2024-03-04T16:33:41.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fac02954d0bf3e9-0065e5f7e5-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:41.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:41.967 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:41.967 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:41.968 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.968 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.968 DEBUG:boto:Method: GET 2024-03-04T16:33:41.968 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.968 DEBUG:boto:Data: 2024-03-04T16:33:41.968 DEBUG:boto:Headers: {} 2024-03-04T16:33:41.968 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:41.968 DEBUG:boto:Port: 8000 2024-03-04T16:33:41.968 DEBUG:boto:Params: {} 2024-03-04T16:33:41.968 DEBUG:boto:Token: None 2024-03-04T16:33:41.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:41 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:41.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M= 2024-03-04T16:33:41.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ovCt7ZSPAs2eorMjWFKS2Rbvz5M=', 'Content-Length': '0'} 2024-03-04T16:33:41.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000871294938d68c33a-0065e5f7e5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.012 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.012 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.012 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:42.012 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:42.013 DEBUG:boto:Method: GET 2024-03-04T16:33:42.013 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:42.013 DEBUG:boto:Data: 2024-03-04T16:33:42.013 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.013 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.013 DEBUG:boto:Params: {} 2024-03-04T16:33:42.013 DEBUG:boto:Token: None 2024-03-04T16:33:42.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:42.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lANjkYbJtGv5YyRZS2HH7Q4X4Fw= 2024-03-04T16:33:42.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lANjkYbJtGv5YyRZS2HH7Q4X4Fw=', 'Content-Length': '0'} 2024-03-04T16:33:42.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ea985417c315a1f-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.056 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.056 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.056 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.057 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.057 DEBUG:boto:Method: GET 2024-03-04T16:33:42.057 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.057 DEBUG:boto:Data: 2024-03-04T16:33:42.057 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.057 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.058 DEBUG:boto:Params: {} 2024-03-04T16:33:42.058 DEBUG:boto:Token: None 2024-03-04T16:33:42.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb383148df796754-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.108 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.109 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.109 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.109 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.109 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.109 DEBUG:boto:Method: GET 2024-03-04T16:33:42.109 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.109 DEBUG:boto:Data: 2024-03-04T16:33:42.109 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.109 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.110 DEBUG:boto:Params: {} 2024-03-04T16:33:42.110 DEBUG:boto:Token: None 2024-03-04T16:33:42.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f8f6d97848bde8e-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.113 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.114 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.114 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.114 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.114 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.114 DEBUG:boto:Data: 2024-03-04T16:33:42.114 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.114 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.114 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.114 DEBUG:boto:Params: {} 2024-03-04T16:33:42.114 DEBUG:boto:Token: None 2024-03-04T16:33:42.115 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.115 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eac7184abefee7fd-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.119 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.119 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.119 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.119 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.119 DEBUG:boto:Data: 2024-03-04T16:33:42.119 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.119 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.119 DEBUG:boto:Params: {} 2024-03-04T16:33:42.120 DEBUG:boto:Token: None 2024-03-04T16:33:42.120 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b60ca9559a527b23-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.124 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.124 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.124 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.124 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.124 DEBUG:boto:Method: GET 2024-03-04T16:33:42.124 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.124 DEBUG:boto:Data: 2024-03-04T16:33:42.124 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.124 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.124 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.125 DEBUG:boto:Params: {} 2024-03-04T16:33:42.125 DEBUG:boto:Token: None 2024-03-04T16:33:42.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.125 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008414666f1e52bc1b-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.127 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.127 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.127 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.128 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.128 DEBUG:boto:Method: GET 2024-03-04T16:33:42.128 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.128 DEBUG:boto:Data: 2024-03-04T16:33:42.128 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.128 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.128 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.128 DEBUG:boto:Params: {} 2024-03-04T16:33:42.128 DEBUG:boto:Token: None 2024-03-04T16:33:42.128 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7074c4da42ac17a-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.172 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.172 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.172 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.173 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.173 DEBUG:boto:Method: GET 2024-03-04T16:33:42.173 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.173 DEBUG:boto:Data: 2024-03-04T16:33:42.173 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.173 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.173 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.173 DEBUG:boto:Params: {} 2024-03-04T16:33:42.173 DEBUG:boto:Token: None 2024-03-04T16:33:42.174 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373be11f9b82ed8e-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.177 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.177 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.177 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.177 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.177 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.177 DEBUG:boto:Method: GET 2024-03-04T16:33:42.178 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.178 DEBUG:boto:Data: 2024-03-04T16:33:42.178 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.178 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.178 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.178 DEBUG:boto:Params: {} 2024-03-04T16:33:42.178 DEBUG:boto:Token: None 2024-03-04T16:33:42.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569088757b9cf1da-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.181 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.181 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.181 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.182 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.182 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.182 DEBUG:boto:Method: GET 2024-03-04T16:33:42.182 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.182 DEBUG:boto:Data: 2024-03-04T16:33:42.182 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.182 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.182 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.182 DEBUG:boto:Params: {} 2024-03-04T16:33:42.182 DEBUG:boto:Token: None 2024-03-04T16:33:42.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.183 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g= 2024-03-04T16:33:42.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g=', 'Content-Length': '0'} 2024-03-04T16:33:42.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001244018594701579-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.228 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.228 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.228 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.228 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.229 DEBUG:boto:Method: GET 2024-03-04T16:33:42.229 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.229 DEBUG:boto:Data: 2024-03-04T16:33:42.229 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.229 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.229 DEBUG:boto:Params: {} 2024-03-04T16:33:42.229 DEBUG:boto:Token: None 2024-03-04T16:33:42.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g= 2024-03-04T16:33:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g=', 'Content-Length': '0'} 2024-03-04T16:33:42.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f81926889607fbb-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.233 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.233 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.233 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.233 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.233 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.234 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.234 DEBUG:boto:Data: 2024-03-04T16:33:42.234 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.234 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.234 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.234 DEBUG:boto:Params: {} 2024-03-04T16:33:42.234 DEBUG:boto:Token: None 2024-03-04T16:33:42.234 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.234 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d4gRPbvTQrpA/q+LrSgnvIm3cd0= 2024-03-04T16:33:42.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d4gRPbvTQrpA/q+LrSgnvIm3cd0=', 'Content-Length': '0'} 2024-03-04T16:33:42.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000098d7b74a912989f-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.237 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.237 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.237 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.237 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.237 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.237 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.237 DEBUG:boto:Data: 2024-03-04T16:33:42.237 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.237 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.237 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.237 DEBUG:boto:Params: {} 2024-03-04T16:33:42.238 DEBUG:boto:Token: None 2024-03-04T16:33:42.238 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.238 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d4gRPbvTQrpA/q+LrSgnvIm3cd0= 2024-03-04T16:33:42.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d4gRPbvTQrpA/q+LrSgnvIm3cd0=', 'Content-Length': '0'} 2024-03-04T16:33:42.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000581e1c00e10c8cc-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.240 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.240 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.241 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.241 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.241 DEBUG:boto:Method: GET 2024-03-04T16:33:42.241 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.241 DEBUG:boto:Data: 2024-03-04T16:33:42.241 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.241 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.241 DEBUG:boto:Params: {} 2024-03-04T16:33:42.241 DEBUG:boto:Token: None 2024-03-04T16:33:42.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g= 2024-03-04T16:33:42.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g=', 'Content-Length': '0'} 2024-03-04T16:33:42.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a655ee200dc76a5-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.244 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.244 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.244 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.244 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.244 DEBUG:boto:Method: GET 2024-03-04T16:33:42.244 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.244 DEBUG:boto:Data: 2024-03-04T16:33:42.244 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.244 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.245 DEBUG:boto:Params: {} 2024-03-04T16:33:42.245 DEBUG:boto:Token: None 2024-03-04T16:33:42.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:42.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g= 2024-03-04T16:33:42.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RemQ/JrBT/q3YdGL7LGjQcpWS9g=', 'Content-Length': '0'} 2024-03-04T16:33:42.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003488b590a46e2673-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.288 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.288 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.288 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.288 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.289 DEBUG:boto:Method: GET 2024-03-04T16:33:42.289 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.289 DEBUG:boto:Data: 2024-03-04T16:33:42.289 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.289 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.289 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.289 DEBUG:boto:Params: {} 2024-03-04T16:33:42.289 DEBUG:boto:Token: None 2024-03-04T16:33:42.289 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000962c5cd09c53eb59-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.293 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.293 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.293 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.293 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.293 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.293 DEBUG:boto:Method: GET 2024-03-04T16:33:42.293 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.294 DEBUG:boto:Data: 2024-03-04T16:33:42.294 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.294 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.294 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.294 DEBUG:boto:Params: {} 2024-03-04T16:33:42.294 DEBUG:boto:Token: None 2024-03-04T16:33:42.294 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002750f675cb860a16-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.340 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.340 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.341 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.341 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.341 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.341 DEBUG:boto:Data: 2024-03-04T16:33:42.341 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.341 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.341 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.341 DEBUG:boto:Params: {} 2024-03-04T16:33:42.341 DEBUG:boto:Token: None 2024-03-04T16:33:42.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055eeecb01a2fd99d-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.345 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.346 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.346 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.346 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.346 DEBUG:boto:Data: 2024-03-04T16:33:42.346 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.346 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.346 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.346 DEBUG:boto:Params: {} 2024-03-04T16:33:42.346 DEBUG:boto:Token: None 2024-03-04T16:33:42.346 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.347 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082af169b5f302b64-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.350 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.350 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.350 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.350 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.351 DEBUG:boto:Method: GET 2024-03-04T16:33:42.351 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.351 DEBUG:boto:Data: 2024-03-04T16:33:42.351 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.351 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.351 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.351 DEBUG:boto:Params: {} 2024-03-04T16:33:42.351 DEBUG:boto:Token: None 2024-03-04T16:33:42.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.351 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005046b33662260a31-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.397 DEBUG:boto:Method: GET 2024-03-04T16:33:42.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.397 DEBUG:boto:Data: 2024-03-04T16:33:42.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.397 DEBUG:boto:Params: {} 2024-03-04T16:33:42.397 DEBUG:boto:Token: None 2024-03-04T16:33:42.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f9654df8ef9154-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.440 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.440 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.440 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.441 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.441 DEBUG:boto:Method: GET 2024-03-04T16:33:42.441 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.441 DEBUG:boto:Data: 2024-03-04T16:33:42.441 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.441 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.441 DEBUG:boto:Params: {} 2024-03-04T16:33:42.441 DEBUG:boto:Token: None 2024-03-04T16:33:42.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b006986641ea150-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.444 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.445 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.445 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.445 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.445 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.445 DEBUG:boto:Method: GET 2024-03-04T16:33:42.445 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.445 DEBUG:boto:Data: 2024-03-04T16:33:42.446 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.446 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.446 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.446 DEBUG:boto:Params: {} 2024-03-04T16:33:42.446 DEBUG:boto:Token: None 2024-03-04T16:33:42.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f310274706c87afa-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.492 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.493 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.494 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.494 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.494 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.494 DEBUG:boto:Method: GET 2024-03-04T16:33:42.494 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.494 DEBUG:boto:Data: 2024-03-04T16:33:42.494 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.494 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.494 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.494 DEBUG:boto:Params: {} 2024-03-04T16:33:42.495 DEBUG:boto:Token: None 2024-03-04T16:33:42.495 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.495 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc= 2024-03-04T16:33:42.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc=', 'Content-Length': '0'} 2024-03-04T16:33:42.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020c718a55b2a339c-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.540 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.540 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.540 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.541 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.541 DEBUG:boto:Method: GET 2024-03-04T16:33:42.541 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.541 DEBUG:boto:Data: 2024-03-04T16:33:42.541 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.541 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.541 DEBUG:boto:Params: {} 2024-03-04T16:33:42.541 DEBUG:boto:Token: None 2024-03-04T16:33:42.541 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc= 2024-03-04T16:33:42.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc=', 'Content-Length': '0'} 2024-03-04T16:33:42.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc6b7faab96adb0-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.544 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.544 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.544 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.545 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.545 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.545 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.545 DEBUG:boto:Data: 2024-03-04T16:33:42.545 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.545 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.545 DEBUG:boto:Params: {} 2024-03-04T16:33:42.545 DEBUG:boto:Token: None 2024-03-04T16:33:42.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6D6GfTfySJrVJQFOp7ju17YU3vQ= 2024-03-04T16:33:42.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6D6GfTfySJrVJQFOp7ju17YU3vQ=', 'Content-Length': '0'} 2024-03-04T16:33:42.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000124c4133cf68c973-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.548 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.548 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.548 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.548 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.548 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.548 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.548 DEBUG:boto:Data: 2024-03-04T16:33:42.548 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.548 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.548 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.548 DEBUG:boto:Params: {} 2024-03-04T16:33:42.549 DEBUG:boto:Token: None 2024-03-04T16:33:42.549 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6D6GfTfySJrVJQFOp7ju17YU3vQ= 2024-03-04T16:33:42.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6D6GfTfySJrVJQFOp7ju17YU3vQ=', 'Content-Length': '0'} 2024-03-04T16:33:42.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000541159eb2e089bc1-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.551 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.551 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.551 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.552 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.552 DEBUG:boto:Method: GET 2024-03-04T16:33:42.552 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.552 DEBUG:boto:Data: 2024-03-04T16:33:42.552 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.552 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.552 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.552 DEBUG:boto:Params: {} 2024-03-04T16:33:42.552 DEBUG:boto:Token: None 2024-03-04T16:33:42.552 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc= 2024-03-04T16:33:42.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc=', 'Content-Length': '0'} 2024-03-04T16:33:42.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc99e0dac28ee58b-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.600 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.600 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.600 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.600 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.601 DEBUG:boto:Method: GET 2024-03-04T16:33:42.601 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.601 DEBUG:boto:Data: 2024-03-04T16:33:42.601 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.601 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.601 DEBUG:boto:Params: {} 2024-03-04T16:33:42.601 DEBUG:boto:Token: None 2024-03-04T16:33:42.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:42.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc= 2024-03-04T16:33:42.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fwQEEOGTyXJbK+ES8R1h6nmvYfc=', 'Content-Length': '0'} 2024-03-04T16:33:42.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ee4aeb193810dcc-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.604 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.604 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.604 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.604 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.604 DEBUG:boto:Method: GET 2024-03-04T16:33:42.604 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.605 DEBUG:boto:Data: 2024-03-04T16:33:42.605 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.605 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.605 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.605 DEBUG:boto:Params: {} 2024-03-04T16:33:42.605 DEBUG:boto:Token: None 2024-03-04T16:33:42.606 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.606 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f219bf3b2875f4d-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.609 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.609 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.609 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.609 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.609 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.610 DEBUG:boto:Method: GET 2024-03-04T16:33:42.610 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.610 DEBUG:boto:Data: 2024-03-04T16:33:42.610 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.610 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.610 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.610 DEBUG:boto:Params: {} 2024-03-04T16:33:42.610 DEBUG:boto:Token: None 2024-03-04T16:33:42.610 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.610 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a08fd3cc06da4a-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.614 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.614 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.614 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.615 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.615 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.615 DEBUG:boto:Data: 2024-03-04T16:33:42.615 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.615 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.615 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.615 DEBUG:boto:Params: {} 2024-03-04T16:33:42.615 DEBUG:boto:Token: None 2024-03-04T16:33:42.615 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.616 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b19b05582a8e56cd-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.619 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.619 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.619 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.620 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.620 DEBUG:boto:Data: 2024-03-04T16:33:42.620 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.620 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.620 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.620 DEBUG:boto:Params: {} 2024-03-04T16:33:42.620 DEBUG:boto:Token: None 2024-03-04T16:33:42.620 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.620 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f0ea3d598124fc7-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.624 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.624 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.624 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.624 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.625 DEBUG:boto:Method: GET 2024-03-04T16:33:42.625 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.625 DEBUG:boto:Data: 2024-03-04T16:33:42.625 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.625 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.625 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.625 DEBUG:boto:Params: {} 2024-03-04T16:33:42.625 DEBUG:boto:Token: None 2024-03-04T16:33:42.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1c08feb4b5294a0-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.672 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.672 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.672 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.673 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.673 DEBUG:boto:Method: GET 2024-03-04T16:33:42.673 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.673 DEBUG:boto:Data: 2024-03-04T16:33:42.673 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.673 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.673 DEBUG:boto:Params: {} 2024-03-04T16:33:42.673 DEBUG:boto:Token: None 2024-03-04T16:33:42.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a992395c57a163b7-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.720 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.720 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.720 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.721 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.721 DEBUG:boto:Method: GET 2024-03-04T16:33:42.721 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.721 DEBUG:boto:Data: 2024-03-04T16:33:42.721 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.721 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.721 DEBUG:boto:Params: {} 2024-03-04T16:33:42.721 DEBUG:boto:Token: None 2024-03-04T16:33:42.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdfe46c809bdab1-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.724 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.725 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.725 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.725 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.725 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.725 DEBUG:boto:Method: GET 2024-03-04T16:33:42.725 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.726 DEBUG:boto:Data: 2024-03-04T16:33:42.726 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.726 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.726 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.726 DEBUG:boto:Params: {} 2024-03-04T16:33:42.726 DEBUG:boto:Token: None 2024-03-04T16:33:42.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5999502f5a852eb-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.776 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.777 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.777 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.777 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.777 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.777 DEBUG:boto:Method: GET 2024-03-04T16:33:42.777 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.777 DEBUG:boto:Data: 2024-03-04T16:33:42.777 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.777 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.777 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.778 DEBUG:boto:Params: {} 2024-03-04T16:33:42.778 DEBUG:boto:Token: None 2024-03-04T16:33:42.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk= 2024-03-04T16:33:42.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk=', 'Content-Length': '0'} 2024-03-04T16:33:42.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c337347c2dd3f892-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.824 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.824 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.824 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.824 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.825 DEBUG:boto:Method: GET 2024-03-04T16:33:42.825 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.825 DEBUG:boto:Data: 2024-03-04T16:33:42.825 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.825 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.825 DEBUG:boto:Params: {} 2024-03-04T16:33:42.825 DEBUG:boto:Token: None 2024-03-04T16:33:42.826 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.826 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk= 2024-03-04T16:33:42.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk=', 'Content-Length': '0'} 2024-03-04T16:33:42.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a6fd5719e8dc831-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.828 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.828 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.828 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.828 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.828 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.829 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.829 DEBUG:boto:Data: 2024-03-04T16:33:42.829 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.829 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.829 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.829 DEBUG:boto:Params: {} 2024-03-04T16:33:42.829 DEBUG:boto:Token: None 2024-03-04T16:33:42.829 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j3B6XgWZlD3k/wMueO5bAreKQqc= 2024-03-04T16:33:42.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j3B6XgWZlD3k/wMueO5bAreKQqc=', 'Content-Length': '0'} 2024-03-04T16:33:42.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c9d5678a8bbfd83-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.831 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.831 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.832 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.832 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.832 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.832 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.832 DEBUG:boto:Data: 2024-03-04T16:33:42.832 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.832 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.832 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.832 DEBUG:boto:Params: {} 2024-03-04T16:33:42.832 DEBUG:boto:Token: None 2024-03-04T16:33:42.833 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j3B6XgWZlD3k/wMueO5bAreKQqc= 2024-03-04T16:33:42.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j3B6XgWZlD3k/wMueO5bAreKQqc=', 'Content-Length': '0'} 2024-03-04T16:33:42.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df18d26430d9b3e1-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.835 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.835 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.835 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.835 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.835 DEBUG:boto:Method: GET 2024-03-04T16:33:42.835 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.835 DEBUG:boto:Data: 2024-03-04T16:33:42.835 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.836 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.836 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.836 DEBUG:boto:Params: {} 2024-03-04T16:33:42.836 DEBUG:boto:Token: None 2024-03-04T16:33:42.836 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.836 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk= 2024-03-04T16:33:42.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk=', 'Content-Length': '0'} 2024-03-04T16:33:42.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e50edad55049fda8-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.884 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.885 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.885 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.885 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.885 DEBUG:boto:Method: GET 2024-03-04T16:33:42.885 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.885 DEBUG:boto:Data: 2024-03-04T16:33:42.885 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.885 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.885 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.885 DEBUG:boto:Params: {} 2024-03-04T16:33:42.886 DEBUG:boto:Token: None 2024-03-04T16:33:42.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:42.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk= 2024-03-04T16:33:42.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rkkVWTzIh8fxfbHEirAASVErqwk=', 'Content-Length': '0'} 2024-03-04T16:33:42.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009486d4d56de52a1b-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.944 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.944 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.944 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.944 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.945 DEBUG:boto:Method: GET 2024-03-04T16:33:42.945 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.945 DEBUG:boto:Data: 2024-03-04T16:33:42.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.945 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.945 DEBUG:boto:Params: {} 2024-03-04T16:33:42.945 DEBUG:boto:Token: None 2024-03-04T16:33:42.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c46d1eaa9e3f76c1-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.949 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.949 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:42.949 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:42.949 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.949 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:42.949 DEBUG:boto:Method: GET 2024-03-04T16:33:42.950 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:42.950 DEBUG:boto:Data: 2024-03-04T16:33:42.950 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.950 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.950 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.950 DEBUG:boto:Params: {} 2024-03-04T16:33:42.950 DEBUG:boto:Token: None 2024-03-04T16:33:42.950 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/ 2024-03-04T16:33:42.950 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410= 2024-03-04T16:33:42.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bIng+Y0s1BbrfpVbGNHV2CrJ410=', 'Content-Length': '0'} 2024-03-04T16:33:42.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649414e459806550-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.953 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:42.953 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.954 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.954 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.954 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.954 DEBUG:boto:Data: 2024-03-04T16:33:42.954 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.954 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.954 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.954 DEBUG:boto:Params: {} 2024-03-04T16:33:42.954 DEBUG:boto:Token: None 2024-03-04T16:33:42.954 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.954 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c30d1b3c16c2902-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.958 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.958 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.958 DEBUG:boto:Method: HEAD 2024-03-04T16:33:42.958 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:42.958 DEBUG:boto:Data: 2024-03-04T16:33:42.958 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.959 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:42.959 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.959 DEBUG:boto:Params: {} 2024-03-04T16:33:42.959 DEBUG:boto:Token: None 2024-03-04T16:33:42.959 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj 2024-03-04T16:33:42.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc= 2024-03-04T16:33:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Pe5tX2AW82wE4wmipRr5InQY+Pc=', 'Content-Length': '0'} 2024-03-04T16:33:42.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4d0e2688840a88c-0065e5f7e6-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:42.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:42.962 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:42.963 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:42.963 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.963 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.963 DEBUG:boto:Method: GET 2024-03-04T16:33:42.963 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.963 DEBUG:boto:Data: 2024-03-04T16:33:42.963 DEBUG:boto:Headers: {} 2024-03-04T16:33:42.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:42.963 DEBUG:boto:Port: 8000 2024-03-04T16:33:42.963 DEBUG:boto:Params: {} 2024-03-04T16:33:42.963 DEBUG:boto:Token: None 2024-03-04T16:33:42.964 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:42 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:42.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg= 2024-03-04T16:33:42.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:42 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5Y8PceaM9C5LDG2F34kSAnvr2gg=', 'Content-Length': '0'} 2024-03-04T16:33:42.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3880b43e335749-0065e5f7e6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:42 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.012 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.012 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.012 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.012 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.013 DEBUG:boto:Method: GET 2024-03-04T16:33:43.013 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.013 DEBUG:boto:Data: 2024-03-04T16:33:43.013 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.013 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.013 DEBUG:boto:Params: {} 2024-03-04T16:33:43.013 DEBUG:boto:Token: None 2024-03-04T16:33:43.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082fe736b929fb17b-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.060 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.060 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.060 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.061 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.061 DEBUG:boto:Method: GET 2024-03-04T16:33:43.061 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.061 DEBUG:boto:Data: 2024-03-04T16:33:43.061 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.061 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:43.061 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.061 DEBUG:boto:Params: {} 2024-03-04T16:33:43.061 DEBUG:boto:Token: None 2024-03-04T16:33:43.062 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.062 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e51b84494ed00e2-0065e5f7e7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.065 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.065 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.065 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.065 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.066 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.066 DEBUG:boto:Method: GET 2024-03-04T16:33:43.066 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.066 DEBUG:boto:Data: 2024-03-04T16:33:43.066 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.066 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.066 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.066 DEBUG:boto:Params: {} 2024-03-04T16:33:43.066 DEBUG:boto:Token: None 2024-03-04T16:33:43.066 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.067 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a4ff0392323f59-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.069 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.069 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone2, test-zone1} 2024-03-04T16:33:43.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone3, test-zone1} 2024-03-04T16:33:43.070 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.070 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.070 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.070 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:43.070 DEBUG:boto:Data: 2024-03-04T16:33:43.070 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.070 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.070 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.070 DEBUG:boto:Params: {} 2024-03-04T16:33:43.071 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:33:43.071 DEBUG:boto:Token: None 2024-03-04T16:33:43.071 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.071 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QjnGMbmb9SC8p9iqX0xsQpfrd/Q= 2024-03-04T16:33:43.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QjnGMbmb9SC8p9iqX0xsQpfrd/Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000345c4234588cb9ae-0065e5f7e7-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.080 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.080 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.080 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.080 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:43.080 DEBUG:boto:Data: 2024-03-04T16:33:43.080 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.080 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.080 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.080 DEBUG:boto:Params: {} 2024-03-04T16:33:43.081 DEBUG:boto:Token: None 2024-03-04T16:33:43.081 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QjnGMbmb9SC8p9iqX0xsQpfrd/Q= 2024-03-04T16:33:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QjnGMbmb9SC8p9iqX0xsQpfrd/Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b639adcab09de5c9-0065e5f7e7-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:33:43.083 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.084 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.084 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.084 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.084 DEBUG:boto:Method: GET 2024-03-04T16:33:43.084 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.084 DEBUG:boto:Data: 2024-03-04T16:33:43.084 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.084 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.084 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.084 DEBUG:boto:Params: {} 2024-03-04T16:33:43.084 DEBUG:boto:Token: None 2024-03-04T16:33:43.085 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/?versions 2024-03-04T16:33:43.085 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU= 2024-03-04T16:33:43.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU=', 'Content-Length': '0'} 2024-03-04T16:33:43.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69db2f8dc2d9d6c-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.088 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.089 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:43.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.092 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:33:43.092 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.092 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.093 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.093 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.093 DEBUG:boto:Method: GET 2024-03-04T16:33:43.093 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.093 DEBUG:boto:Data: 2024-03-04T16:33:43.093 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.093 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.093 DEBUG:boto:Params: {} 2024-03-04T16:33:43.093 DEBUG:boto:Token: None 2024-03-04T16:33:43.094 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/?versions 2024-03-04T16:33:43.094 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU= 2024-03-04T16:33:43.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU=', 'Content-Length': '0'} 2024-03-04T16:33:43.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd9f0b6d72d5c9f7-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.097 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.098 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:43.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:43.101 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.101 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.101 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.101 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.101 DEBUG:boto:Method: GET 2024-03-04T16:33:43.101 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.102 DEBUG:boto:Data: 2024-03-04T16:33:43.102 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.102 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.102 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.102 DEBUG:boto:Params: {} 2024-03-04T16:33:43.102 DEBUG:boto:Token: None 2024-03-04T16:33:43.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/?versions 2024-03-04T16:33:43.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU= 2024-03-04T16:33:43.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU=', 'Content-Length': '0'} 2024-03-04T16:33:43.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488018e001c321b6-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.106 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.107 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.108 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.108 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.108 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.108 DEBUG:boto:Method: GET 2024-03-04T16:33:43.108 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:43.108 DEBUG:boto:Data: 2024-03-04T16:33:43.108 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.108 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.108 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.109 DEBUG:boto:Params: {} 2024-03-04T16:33:43.109 DEBUG:boto:Token: None 2024-03-04T16:33:43.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/?versions 2024-03-04T16:33:43.109 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU= 2024-03-04T16:33:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RGcQS4o++IUOaqbtbK+IvB5oWRU=', 'Content-Length': '0'} 2024-03-04T16:33:43.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093111ba6c41ad5ab-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.112 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:43.114 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.114 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.114 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.114 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.115 DEBUG:boto:Method: GET 2024-03-04T16:33:43.115 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.115 DEBUG:boto:Data: 2024-03-04T16:33:43.115 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.115 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.115 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.115 DEBUG:boto:Params: {} 2024-03-04T16:33:43.115 DEBUG:boto:Token: None 2024-03-04T16:33:43.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.115 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d33534ab475d448-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.118 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.118 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.118 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.118 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.118 DEBUG:boto:Method: GET 2024-03-04T16:33:43.118 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.118 DEBUG:boto:Data: 2024-03-04T16:33:43.118 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.118 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.118 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.119 DEBUG:boto:Params: {} 2024-03-04T16:33:43.119 DEBUG:boto:Token: None 2024-03-04T16:33:43.119 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.119 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ccb85ecb85ec5c2-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.121 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.121 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.121 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.121 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.121 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.121 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.121 DEBUG:boto:Data: 2024-03-04T16:33:43.121 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.122 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.122 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.122 DEBUG:boto:Params: {} 2024-03-04T16:33:43.122 DEBUG:boto:Token: None 2024-03-04T16:33:43.122 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AhtXlwuTrE4ffoLjM6KV8AV+5sY= 2024-03-04T16:33:43.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AhtXlwuTrE4ffoLjM6KV8AV+5sY=', 'Content-Length': '0'} 2024-03-04T16:33:43.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000979210518e94d312-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.124 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.124 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.124 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.125 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.125 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.125 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.125 DEBUG:boto:Data: 2024-03-04T16:33:43.125 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.125 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.125 DEBUG:boto:Params: {} 2024-03-04T16:33:43.125 DEBUG:boto:Token: None 2024-03-04T16:33:43.125 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AhtXlwuTrE4ffoLjM6KV8AV+5sY= 2024-03-04T16:33:43.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AhtXlwuTrE4ffoLjM6KV8AV+5sY=', 'Content-Length': '0'} 2024-03-04T16:33:43.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050d4f5c1d752a996-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:43.128 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.128 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.128 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.128 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.128 DEBUG:boto:Method: GET 2024-03-04T16:33:43.128 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.129 DEBUG:boto:Data: 2024-03-04T16:33:43.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.129 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.129 DEBUG:boto:Params: {} 2024-03-04T16:33:43.129 DEBUG:boto:Token: None 2024-03-04T16:33:43.129 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.129 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c693023ca0c1e43-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.131 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.131 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.131 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.131 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.132 DEBUG:boto:Method: GET 2024-03-04T16:33:43.132 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.132 DEBUG:boto:Data: 2024-03-04T16:33:43.132 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.132 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.132 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.132 DEBUG:boto:Params: {} 2024-03-04T16:33:43.132 DEBUG:boto:Token: None 2024-03-04T16:33:43.133 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.133 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ddff0fb1b0d7007-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.135 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.135 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.135 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.135 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.135 DEBUG:boto:Method: GET 2024-03-04T16:33:43.135 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.135 DEBUG:boto:Data: 2024-03-04T16:33:43.135 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.135 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.136 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.136 DEBUG:boto:Params: {} 2024-03-04T16:33:43.136 DEBUG:boto:Token: None 2024-03-04T16:33:43.136 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.136 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c300c35dc234aa09-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.139 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.139 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.139 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.139 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.139 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.140 DEBUG:boto:Method: GET 2024-03-04T16:33:43.140 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.140 DEBUG:boto:Data: 2024-03-04T16:33:43.140 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.140 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.140 DEBUG:boto:Params: {} 2024-03-04T16:33:43.140 DEBUG:boto:Token: None 2024-03-04T16:33:43.140 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000874abba559a20302-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.188 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.189 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.189 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.189 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.189 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:43.189 DEBUG:boto:Data: 2024-03-04T16:33:43.189 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.189 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.189 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.189 DEBUG:boto:Params: {} 2024-03-04T16:33:43.190 DEBUG:boto:Token: None 2024-03-04T16:33:43.190 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy 2024-03-04T16:33:43.190 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Wg8+TI+M/418nKetZruGRqfklac= 2024-03-04T16:33:43.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Wg8+TI+M/418nKetZruGRqfklac=', 'Content-Length': '0'} 2024-03-04T16:33:43.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020f8464213b8ba31-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.193 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.193 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.193 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.193 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:43.193 DEBUG:boto:Data: 2024-03-04T16:33:43.193 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.193 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.193 DEBUG:boto:Params: {} 2024-03-04T16:33:43.194 DEBUG:boto:Token: None 2024-03-04T16:33:43.194 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy 2024-03-04T16:33:43.194 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Wg8+TI+M/418nKetZruGRqfklac= 2024-03-04T16:33:43.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Wg8+TI+M/418nKetZruGRqfklac=', 'Content-Length': '0'} 2024-03-04T16:33:43.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dac556de388adf1-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:43.196 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.196 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.196 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.197 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.197 DEBUG:boto:Method: GET 2024-03-04T16:33:43.197 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.197 DEBUG:boto:Data: 2024-03-04T16:33:43.197 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.197 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.197 DEBUG:boto:Params: {} 2024-03-04T16:33:43.197 DEBUG:boto:Token: None 2024-03-04T16:33:43.197 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006673cbbbdebb8807-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.199 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:43.199 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:43.200 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.200 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.200 DEBUG:boto:Method: GET 2024-03-04T16:33:43.200 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.200 DEBUG:boto:Data: 2024-03-04T16:33:43.200 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.200 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.200 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.200 DEBUG:boto:Params: {} 2024-03-04T16:33:43.200 DEBUG:boto:Token: None 2024-03-04T16:33:43.201 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:43.201 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ= 2024-03-04T16:33:43.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FkSHjNYGvVlruPJT0urEVGGs8HQ=', 'Content-Length': '0'} 2024-03-04T16:33:43.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef0b1013c2706777-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.203 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.203 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.203 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.203 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.203 DEBUG:boto:Method: GET 2024-03-04T16:33:43.203 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.204 DEBUG:boto:Data: 2024-03-04T16:33:43.204 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.204 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.204 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.204 DEBUG:boto:Params: {} 2024-03-04T16:33:43.204 DEBUG:boto:Token: None 2024-03-04T16:33:43.204 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa9bb8dd00cadb4-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.252 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.253 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.253 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.253 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.253 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.253 DEBUG:boto:Method: GET 2024-03-04T16:33:43.253 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:43.253 DEBUG:boto:Data: 2024-03-04T16:33:43.253 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.253 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.253 DEBUG:boto:Params: {} 2024-03-04T16:33:43.254 DEBUG:boto:Token: None 2024-03-04T16:33:43.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036cfc6042a644652-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.256 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.257 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.257 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.257 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.257 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.257 DEBUG:boto:Method: GET 2024-03-04T16:33:43.257 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.257 DEBUG:boto:Data: 2024-03-04T16:33:43.258 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.258 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.258 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.258 DEBUG:boto:Params: {} 2024-03-04T16:33:43.258 DEBUG:boto:Token: None 2024-03-04T16:33:43.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001129d02d5b9dc8e8-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.260 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.260 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.261 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.261 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.261 DEBUG:boto:Method: GET 2024-03-04T16:33:43.261 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.261 DEBUG:boto:Data: 2024-03-04T16:33:43.261 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.261 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.261 DEBUG:boto:Params: {} 2024-03-04T16:33:43.261 DEBUG:boto:Token: None 2024-03-04T16:33:43.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.262 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e449ced0c22e3611-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.264 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.264 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.264 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.264 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.264 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.264 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.264 DEBUG:boto:Data: 2024-03-04T16:33:43.264 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.264 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.264 DEBUG:boto:Params: {} 2024-03-04T16:33:43.265 DEBUG:boto:Token: None 2024-03-04T16:33:43.265 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.265 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AQ/UTVJtBVENnMlfsqvlOoMoccw= 2024-03-04T16:33:43.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AQ/UTVJtBVENnMlfsqvlOoMoccw=', 'Content-Length': '0'} 2024-03-04T16:33:43.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000938860c16ef2e93-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.267 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.267 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.267 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.267 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.267 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.267 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.267 DEBUG:boto:Data: 2024-03-04T16:33:43.267 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.268 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.268 DEBUG:boto:Params: {} 2024-03-04T16:33:43.268 DEBUG:boto:Token: None 2024-03-04T16:33:43.268 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.268 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AQ/UTVJtBVENnMlfsqvlOoMoccw= 2024-03-04T16:33:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AQ/UTVJtBVENnMlfsqvlOoMoccw=', 'Content-Length': '0'} 2024-03-04T16:33:43.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7a0b6f9321a0eb3-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.270 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.270 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.271 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.271 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.271 DEBUG:boto:Method: GET 2024-03-04T16:33:43.271 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.271 DEBUG:boto:Data: 2024-03-04T16:33:43.271 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.271 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.271 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.271 DEBUG:boto:Params: {} 2024-03-04T16:33:43.272 DEBUG:boto:Token: None 2024-03-04T16:33:43.272 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.272 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a03acc1b67738258-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.274 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.274 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.274 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.274 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.274 DEBUG:boto:Method: GET 2024-03-04T16:33:43.274 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.274 DEBUG:boto:Data: 2024-03-04T16:33:43.274 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.274 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.274 DEBUG:boto:Params: {} 2024-03-04T16:33:43.275 DEBUG:boto:Token: None 2024-03-04T16:33:43.275 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064332304a1b51bd2-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.320 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.320 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.320 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.321 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.321 DEBUG:boto:Method: GET 2024-03-04T16:33:43.321 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.321 DEBUG:boto:Data: 2024-03-04T16:33:43.321 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.321 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.321 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.321 DEBUG:boto:Params: {} 2024-03-04T16:33:43.321 DEBUG:boto:Token: None 2024-03-04T16:33:43.322 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.322 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dea2d86bafe00d4-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.324 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.325 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.325 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.325 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.325 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.325 DEBUG:boto:Method: GET 2024-03-04T16:33:43.325 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.325 DEBUG:boto:Data: 2024-03-04T16:33:43.325 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.326 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.326 DEBUG:boto:Params: {} 2024-03-04T16:33:43.326 DEBUG:boto:Token: None 2024-03-04T16:33:43.327 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.327 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a49e95eabcef28-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.372 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.372 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.373 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.373 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.373 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.373 DEBUG:boto:Data: 2024-03-04T16:33:43.373 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.373 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.373 DEBUG:boto:Params: {} 2024-03-04T16:33:43.373 DEBUG:boto:Token: None 2024-03-04T16:33:43.374 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.374 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000391c710c9012d0fd-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.377 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.377 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.377 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.377 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.378 DEBUG:boto:Data: 2024-03-04T16:33:43.378 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.378 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.378 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.378 DEBUG:boto:Params: {} 2024-03-04T16:33:43.378 DEBUG:boto:Token: None 2024-03-04T16:33:43.378 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2d14a78499efb89-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.381 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.382 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.382 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.382 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.382 DEBUG:boto:Method: GET 2024-03-04T16:33:43.382 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.382 DEBUG:boto:Data: 2024-03-04T16:33:43.382 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.382 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.382 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.382 DEBUG:boto:Params: {} 2024-03-04T16:33:43.383 DEBUG:boto:Token: None 2024-03-04T16:33:43.383 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.383 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080ff4f4def63e4c2-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.428 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.428 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.428 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.428 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.429 DEBUG:boto:Method: GET 2024-03-04T16:33:43.429 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.429 DEBUG:boto:Data: 2024-03-04T16:33:43.429 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.429 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.429 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.429 DEBUG:boto:Params: {} 2024-03-04T16:33:43.429 DEBUG:boto:Token: None 2024-03-04T16:33:43.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e484b55af72db9d7-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.476 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.476 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.477 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.477 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.477 DEBUG:boto:Method: GET 2024-03-04T16:33:43.477 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.477 DEBUG:boto:Data: 2024-03-04T16:33:43.477 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.477 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.477 DEBUG:boto:Params: {} 2024-03-04T16:33:43.478 DEBUG:boto:Token: None 2024-03-04T16:33:43.478 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007485721ef7fa99b9-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.481 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.484 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.484 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.484 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.484 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.484 DEBUG:boto:Method: GET 2024-03-04T16:33:43.484 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.485 DEBUG:boto:Data: 2024-03-04T16:33:43.485 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.485 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.485 DEBUG:boto:Params: {} 2024-03-04T16:33:43.485 DEBUG:boto:Token: None 2024-03-04T16:33:43.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.485 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0fd6471f3a873c0-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.529 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.530 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.530 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.530 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.530 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.530 DEBUG:boto:Method: GET 2024-03-04T16:33:43.530 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.531 DEBUG:boto:Data: 2024-03-04T16:33:43.531 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.531 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.531 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.531 DEBUG:boto:Params: {} 2024-03-04T16:33:43.531 DEBUG:boto:Token: None 2024-03-04T16:33:43.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo= 2024-03-04T16:33:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo=', 'Content-Length': '0'} 2024-03-04T16:33:43.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e37f5b3bee8309e-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.534 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.534 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.534 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.534 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.534 DEBUG:boto:Method: GET 2024-03-04T16:33:43.534 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.534 DEBUG:boto:Data: 2024-03-04T16:33:43.534 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.534 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.535 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.535 DEBUG:boto:Params: {} 2024-03-04T16:33:43.535 DEBUG:boto:Token: None 2024-03-04T16:33:43.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.535 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo= 2024-03-04T16:33:43.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo=', 'Content-Length': '0'} 2024-03-04T16:33:43.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012908613d2635279-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.537 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.538 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.538 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.538 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.538 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.538 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.538 DEBUG:boto:Data: 2024-03-04T16:33:43.538 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.539 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.539 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.539 DEBUG:boto:Params: {} 2024-03-04T16:33:43.539 DEBUG:boto:Token: None 2024-03-04T16:33:43.539 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lYIUK79fIcqyNM84hD29Q0I1fBY= 2024-03-04T16:33:43.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lYIUK79fIcqyNM84hD29Q0I1fBY=', 'Content-Length': '0'} 2024-03-04T16:33:43.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ed8204582ff5310-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.541 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.541 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.541 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.541 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.541 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.541 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.541 DEBUG:boto:Data: 2024-03-04T16:33:43.541 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.542 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.542 DEBUG:boto:Params: {} 2024-03-04T16:33:43.542 DEBUG:boto:Token: None 2024-03-04T16:33:43.542 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lYIUK79fIcqyNM84hD29Q0I1fBY= 2024-03-04T16:33:43.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lYIUK79fIcqyNM84hD29Q0I1fBY=', 'Content-Length': '0'} 2024-03-04T16:33:43.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000907f39f15f6fb411-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.544 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.544 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.544 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.545 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.545 DEBUG:boto:Method: GET 2024-03-04T16:33:43.545 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.545 DEBUG:boto:Data: 2024-03-04T16:33:43.545 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.545 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.545 DEBUG:boto:Params: {} 2024-03-04T16:33:43.545 DEBUG:boto:Token: None 2024-03-04T16:33:43.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.545 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo= 2024-03-04T16:33:43.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo=', 'Content-Length': '0'} 2024-03-04T16:33:43.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ae19551b4a8bed-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.547 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.547 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.547 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.548 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.548 DEBUG:boto:Method: GET 2024-03-04T16:33:43.548 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.548 DEBUG:boto:Data: 2024-03-04T16:33:43.548 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.549 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.549 DEBUG:boto:Params: {} 2024-03-04T16:33:43.549 DEBUG:boto:Token: None 2024-03-04T16:33:43.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:43.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo= 2024-03-04T16:33:43.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T2x2tZWhJOyacGpiCsSRISBZsAo=', 'Content-Length': '0'} 2024-03-04T16:33:43.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d44db48f21f86d6-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.592 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.592 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.593 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.593 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.593 DEBUG:boto:Method: GET 2024-03-04T16:33:43.593 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.593 DEBUG:boto:Data: 2024-03-04T16:33:43.593 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.593 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.593 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.593 DEBUG:boto:Params: {} 2024-03-04T16:33:43.594 DEBUG:boto:Token: None 2024-03-04T16:33:43.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9ea850d3ff8317-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.596 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.597 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.597 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.597 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.597 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.597 DEBUG:boto:Method: GET 2024-03-04T16:33:43.597 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.598 DEBUG:boto:Data: 2024-03-04T16:33:43.598 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.598 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.598 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.598 DEBUG:boto:Params: {} 2024-03-04T16:33:43.598 DEBUG:boto:Token: None 2024-03-04T16:33:43.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029c933f289bec9c1-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.648 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.649 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.649 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.649 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.649 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.649 DEBUG:boto:Data: 2024-03-04T16:33:43.649 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.649 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.649 DEBUG:boto:Params: {} 2024-03-04T16:33:43.650 DEBUG:boto:Token: None 2024-03-04T16:33:43.650 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000135cd6949aa459ef-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.653 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.653 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.654 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.654 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.654 DEBUG:boto:Data: 2024-03-04T16:33:43.654 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.654 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.654 DEBUG:boto:Params: {} 2024-03-04T16:33:43.654 DEBUG:boto:Token: None 2024-03-04T16:33:43.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.655 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f530aa89c759b787-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.659 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.659 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.659 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.659 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.659 DEBUG:boto:Method: GET 2024-03-04T16:33:43.659 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.659 DEBUG:boto:Data: 2024-03-04T16:33:43.659 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.659 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.659 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.660 DEBUG:boto:Params: {} 2024-03-04T16:33:43.660 DEBUG:boto:Token: None 2024-03-04T16:33:43.660 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000183c4ba9dc2637a6-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.662 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.662 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.662 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.662 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.662 DEBUG:boto:Method: GET 2024-03-04T16:33:43.662 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.663 DEBUG:boto:Data: 2024-03-04T16:33:43.663 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.663 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.663 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.663 DEBUG:boto:Params: {} 2024-03-04T16:33:43.663 DEBUG:boto:Token: None 2024-03-04T16:33:43.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.663 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d50bd6e5a466f9db-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.708 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.708 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.709 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.709 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.709 DEBUG:boto:Method: GET 2024-03-04T16:33:43.709 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.709 DEBUG:boto:Data: 2024-03-04T16:33:43.709 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.709 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.709 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.709 DEBUG:boto:Params: {} 2024-03-04T16:33:43.709 DEBUG:boto:Token: None 2024-03-04T16:33:43.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d3abbde811d8972-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.713 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.713 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.713 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.713 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.713 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.714 DEBUG:boto:Method: GET 2024-03-04T16:33:43.714 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.714 DEBUG:boto:Data: 2024-03-04T16:33:43.714 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.714 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.714 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.714 DEBUG:boto:Params: {} 2024-03-04T16:33:43.714 DEBUG:boto:Token: None 2024-03-04T16:33:43.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4cccbaf8f9a00e-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.717 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.718 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.718 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.718 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.718 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.718 DEBUG:boto:Method: GET 2024-03-04T16:33:43.718 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.718 DEBUG:boto:Data: 2024-03-04T16:33:43.719 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.719 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.719 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.719 DEBUG:boto:Params: {} 2024-03-04T16:33:43.719 DEBUG:boto:Token: None 2024-03-04T16:33:43.719 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.719 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0= 2024-03-04T16:33:43.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0=', 'Content-Length': '0'} 2024-03-04T16:33:43.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a29dddcd4a24b83f-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.768 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.768 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.768 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.769 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.769 DEBUG:boto:Method: GET 2024-03-04T16:33:43.769 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.769 DEBUG:boto:Data: 2024-03-04T16:33:43.769 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.769 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.769 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.769 DEBUG:boto:Params: {} 2024-03-04T16:33:43.769 DEBUG:boto:Token: None 2024-03-04T16:33:43.769 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.770 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0= 2024-03-04T16:33:43.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0=', 'Content-Length': '0'} 2024-03-04T16:33:43.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcca79422e97ea28-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.772 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.772 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.772 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.772 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.772 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.773 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.773 DEBUG:boto:Data: 2024-03-04T16:33:43.773 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.773 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.773 DEBUG:boto:Params: {} 2024-03-04T16:33:43.773 DEBUG:boto:Token: None 2024-03-04T16:33:43.773 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.773 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:103tLMRgltzyoFd1/7CBM3AR2rA= 2024-03-04T16:33:43.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:103tLMRgltzyoFd1/7CBM3AR2rA=', 'Content-Length': '0'} 2024-03-04T16:33:43.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055a6993982921592-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.775 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.775 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.776 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.776 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.776 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.776 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.776 DEBUG:boto:Data: 2024-03-04T16:33:43.776 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.776 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.776 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.776 DEBUG:boto:Params: {} 2024-03-04T16:33:43.776 DEBUG:boto:Token: None 2024-03-04T16:33:43.777 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:103tLMRgltzyoFd1/7CBM3AR2rA= 2024-03-04T16:33:43.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:103tLMRgltzyoFd1/7CBM3AR2rA=', 'Content-Length': '0'} 2024-03-04T16:33:43.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031f2a039982073a6-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.779 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.779 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.780 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.780 DEBUG:boto:Method: GET 2024-03-04T16:33:43.780 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.780 DEBUG:boto:Data: 2024-03-04T16:33:43.780 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.780 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.780 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.780 DEBUG:boto:Params: {} 2024-03-04T16:33:43.780 DEBUG:boto:Token: None 2024-03-04T16:33:43.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.781 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0= 2024-03-04T16:33:43.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0=', 'Content-Length': '0'} 2024-03-04T16:33:43.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7699db0f03ce7ab-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.783 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.783 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.783 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.783 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.783 DEBUG:boto:Method: GET 2024-03-04T16:33:43.783 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.783 DEBUG:boto:Data: 2024-03-04T16:33:43.783 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.783 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.783 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.783 DEBUG:boto:Params: {} 2024-03-04T16:33:43.784 DEBUG:boto:Token: None 2024-03-04T16:33:43.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:43.784 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0= 2024-03-04T16:33:43.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2OT/gbwiljK179gVoHk3voFORk0=', 'Content-Length': '0'} 2024-03-04T16:33:43.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c59cb333cf028292-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.786 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.786 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.786 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.786 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.786 DEBUG:boto:Method: GET 2024-03-04T16:33:43.786 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.786 DEBUG:boto:Data: 2024-03-04T16:33:43.786 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.787 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.787 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.787 DEBUG:boto:Params: {} 2024-03-04T16:33:43.787 DEBUG:boto:Token: None 2024-03-04T16:33:43.787 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.787 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb109f39f44c7804-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.790 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.791 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.791 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.791 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.791 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.791 DEBUG:boto:Method: GET 2024-03-04T16:33:43.791 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.791 DEBUG:boto:Data: 2024-03-04T16:33:43.791 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.792 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.792 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.792 DEBUG:boto:Params: {} 2024-03-04T16:33:43.792 DEBUG:boto:Token: None 2024-03-04T16:33:43.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed49267f05f5466-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.795 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.795 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.795 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.795 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.795 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.795 DEBUG:boto:Data: 2024-03-04T16:33:43.795 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.796 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.796 DEBUG:boto:Params: {} 2024-03-04T16:33:43.796 DEBUG:boto:Token: None 2024-03-04T16:33:43.796 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.796 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000341194d38c3df857-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.799 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.799 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.799 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.799 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:43.799 DEBUG:boto:Data: 2024-03-04T16:33:43.799 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.800 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.800 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.800 DEBUG:boto:Params: {} 2024-03-04T16:33:43.800 DEBUG:boto:Token: None 2024-03-04T16:33:43.800 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj 2024-03-04T16:33:43.800 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA= 2024-03-04T16:33:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ohJl9G68ZwElMx7/gXsdkNiNjrA=', 'Content-Length': '0'} 2024-03-04T16:33:43.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ebb3e94b326b73e-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.803 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.803 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.803 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.803 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.804 DEBUG:boto:Method: GET 2024-03-04T16:33:43.804 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.804 DEBUG:boto:Data: 2024-03-04T16:33:43.804 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.804 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.804 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.804 DEBUG:boto:Params: {} 2024-03-04T16:33:43.804 DEBUG:boto:Token: None 2024-03-04T16:33:43.804 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.804 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7a0715dae66a8b7-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.807 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.807 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.807 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.807 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.807 DEBUG:boto:Method: GET 2024-03-04T16:33:43.807 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.807 DEBUG:boto:Data: 2024-03-04T16:33:43.807 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.807 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.807 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.807 DEBUG:boto:Params: {} 2024-03-04T16:33:43.808 DEBUG:boto:Token: None 2024-03-04T16:33:43.808 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:43.808 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34= 2024-03-04T16:33:43.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DAJIiHnZYFCDwsanGumRL0KsO34=', 'Content-Length': '0'} 2024-03-04T16:33:43.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704927f7649aef76-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.856 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.856 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.856 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.857 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.857 DEBUG:boto:Method: GET 2024-03-04T16:33:43.857 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.857 DEBUG:boto:Data: 2024-03-04T16:33:43.857 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.857 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.857 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.857 DEBUG:boto:Params: {} 2024-03-04T16:33:43.857 DEBUG:boto:Token: None 2024-03-04T16:33:43.858 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.858 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48e90f825689cac-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.860 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.861 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:43.861 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:43.861 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.861 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:43.861 DEBUG:boto:Method: GET 2024-03-04T16:33:43.861 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:43.861 DEBUG:boto:Data: 2024-03-04T16:33:43.862 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.862 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.862 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.862 DEBUG:boto:Params: {} 2024-03-04T16:33:43.862 DEBUG:boto:Token: None 2024-03-04T16:33:43.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/ 2024-03-04T16:33:43.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q= 2024-03-04T16:33:43.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:npWvfO4JK0oTxOvLVc9/8vqEy4Q=', 'Content-Length': '0'} 2024-03-04T16:33:43.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388c6b33b3735655-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.908 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:43.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.909 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.909 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.909 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.909 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.909 DEBUG:boto:Method: GET 2024-03-04T16:33:43.909 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.910 DEBUG:boto:Data: 2024-03-04T16:33:43.910 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.910 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.910 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.910 DEBUG:boto:Params: {} 2024-03-04T16:33:43.910 DEBUG:boto:Token: None 2024-03-04T16:33:43.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.911 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU= 2024-03-04T16:33:43.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU=', 'Content-Length': '0'} 2024-03-04T16:33:43.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbe9688341f7550b-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.913 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.913 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.913 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.913 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.913 DEBUG:boto:Method: GET 2024-03-04T16:33:43.914 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.914 DEBUG:boto:Data: 2024-03-04T16:33:43.914 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.914 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.914 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.914 DEBUG:boto:Params: {} 2024-03-04T16:33:43.914 DEBUG:boto:Token: None 2024-03-04T16:33:43.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.915 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU= 2024-03-04T16:33:43.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU=', 'Content-Length': '0'} 2024-03-04T16:33:43.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a006e40dc8bd02b-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.917 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.917 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.917 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.917 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.917 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.917 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.917 DEBUG:boto:Data: 2024-03-04T16:33:43.918 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.918 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.918 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.918 DEBUG:boto:Params: {} 2024-03-04T16:33:43.918 DEBUG:boto:Token: None 2024-03-04T16:33:43.918 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.918 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yZQ88jpxvwlkaEXUvaWC0LArBdE= 2024-03-04T16:33:43.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yZQ88jpxvwlkaEXUvaWC0LArBdE=', 'Content-Length': '0'} 2024-03-04T16:33:43.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6f36b5db2519025-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.920 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.920 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.920 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.920 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.921 DEBUG:boto:Method: HEAD 2024-03-04T16:33:43.921 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.921 DEBUG:boto:Data: 2024-03-04T16:33:43.921 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.921 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.921 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.921 DEBUG:boto:Params: {} 2024-03-04T16:33:43.921 DEBUG:boto:Token: None 2024-03-04T16:33:43.921 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.922 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yZQ88jpxvwlkaEXUvaWC0LArBdE= 2024-03-04T16:33:43.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yZQ88jpxvwlkaEXUvaWC0LArBdE=', 'Content-Length': '0'} 2024-03-04T16:33:43.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e83964ca7f33a35-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:43.924 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.924 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.924 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.924 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.924 DEBUG:boto:Method: GET 2024-03-04T16:33:43.924 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.924 DEBUG:boto:Data: 2024-03-04T16:33:43.924 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:43.925 DEBUG:boto:Port: 8001 2024-03-04T16:33:43.925 DEBUG:boto:Params: {} 2024-03-04T16:33:43.925 DEBUG:boto:Token: None 2024-03-04T16:33:43.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.925 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU= 2024-03-04T16:33:43.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU=', 'Content-Length': '0'} 2024-03-04T16:33:43.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000081ecd269be24a9b-0065e5f7e7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:43.968 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:43.968 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:43.968 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.968 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.969 DEBUG:boto:Method: GET 2024-03-04T16:33:43.969 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.969 DEBUG:boto:Data: 2024-03-04T16:33:43.969 DEBUG:boto:Headers: {} 2024-03-04T16:33:43.969 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:43.969 DEBUG:boto:Port: 8000 2024-03-04T16:33:43.969 DEBUG:boto:Params: {} 2024-03-04T16:33:43.969 DEBUG:boto:Token: None 2024-03-04T16:33:43.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:43 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:43.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU= 2024-03-04T16:33:43.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:43 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jIV+Y8NgO4mgGruE8DnVDBzxrqU=', 'Content-Length': '0'} 2024-03-04T16:33:43.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b0ce9cfad61c5fe-0065e5f7e7-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:43 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.012 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.012 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.013 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.013 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.013 DEBUG:boto:Method: GET 2024-03-04T16:33:44.013 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.013 DEBUG:boto:Data: 2024-03-04T16:33:44.013 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.013 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.013 DEBUG:boto:Params: {} 2024-03-04T16:33:44.014 DEBUG:boto:Token: None 2024-03-04T16:33:44.014 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005565039260085e09-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.017 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.017 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.018 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.018 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.018 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.018 DEBUG:boto:Method: GET 2024-03-04T16:33:44.018 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.018 DEBUG:boto:Data: 2024-03-04T16:33:44.018 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.018 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.018 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.018 DEBUG:boto:Params: {} 2024-03-04T16:33:44.019 DEBUG:boto:Token: None 2024-03-04T16:33:44.019 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.019 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b495d338df450c-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.064 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.065 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.065 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.065 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.065 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.065 DEBUG:boto:Data: 2024-03-04T16:33:44.065 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.065 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.066 DEBUG:boto:Params: {} 2024-03-04T16:33:44.066 DEBUG:boto:Token: None 2024-03-04T16:33:44.066 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db6e318cd138eb53-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.069 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.069 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.070 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.070 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.070 DEBUG:boto:Data: 2024-03-04T16:33:44.070 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.070 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.070 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.070 DEBUG:boto:Params: {} 2024-03-04T16:33:44.070 DEBUG:boto:Token: None 2024-03-04T16:33:44.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.071 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027e93fd16515534b-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.074 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.074 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.074 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.074 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.074 DEBUG:boto:Method: GET 2024-03-04T16:33:44.075 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.075 DEBUG:boto:Data: 2024-03-04T16:33:44.075 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.075 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.075 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.075 DEBUG:boto:Params: {} 2024-03-04T16:33:44.075 DEBUG:boto:Token: None 2024-03-04T16:33:44.075 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.076 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6941f4e12d8bf03-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.078 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.078 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.078 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.078 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.079 DEBUG:boto:Method: GET 2024-03-04T16:33:44.079 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.079 DEBUG:boto:Data: 2024-03-04T16:33:44.079 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.079 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.079 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.079 DEBUG:boto:Params: {} 2024-03-04T16:33:44.079 DEBUG:boto:Token: None 2024-03-04T16:33:44.080 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002778609282378da1-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.124 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.124 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.124 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.124 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.125 DEBUG:boto:Method: GET 2024-03-04T16:33:44.125 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.125 DEBUG:boto:Data: 2024-03-04T16:33:44.125 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.125 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.125 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.125 DEBUG:boto:Params: {} 2024-03-04T16:33:44.125 DEBUG:boto:Token: None 2024-03-04T16:33:44.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ddce5cd2b795f6-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.128 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.129 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.129 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.129 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.129 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.129 DEBUG:boto:Method: GET 2024-03-04T16:33:44.129 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.129 DEBUG:boto:Data: 2024-03-04T16:33:44.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.130 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.130 DEBUG:boto:Params: {} 2024-03-04T16:33:44.130 DEBUG:boto:Token: None 2024-03-04T16:33:44.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001f937cb999786ff-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.176 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.177 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.177 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.177 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.177 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.177 DEBUG:boto:Method: GET 2024-03-04T16:33:44.177 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.177 DEBUG:boto:Data: 2024-03-04T16:33:44.177 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.178 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.178 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.178 DEBUG:boto:Params: {} 2024-03-04T16:33:44.178 DEBUG:boto:Token: None 2024-03-04T16:33:44.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI= 2024-03-04T16:33:44.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI=', 'Content-Length': '0'} 2024-03-04T16:33:44.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000343618abc643de92-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.181 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.181 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.181 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.181 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.181 DEBUG:boto:Method: GET 2024-03-04T16:33:44.181 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.181 DEBUG:boto:Data: 2024-03-04T16:33:44.181 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.181 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.182 DEBUG:boto:Params: {} 2024-03-04T16:33:44.182 DEBUG:boto:Token: None 2024-03-04T16:33:44.182 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI= 2024-03-04T16:33:44.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI=', 'Content-Length': '0'} 2024-03-04T16:33:44.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9d376326c9fd47d-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.184 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.184 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.184 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.184 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.184 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.184 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.184 DEBUG:boto:Data: 2024-03-04T16:33:44.185 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.185 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.185 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.185 DEBUG:boto:Params: {} 2024-03-04T16:33:44.185 DEBUG:boto:Token: None 2024-03-04T16:33:44.185 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.185 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ON/YpS7qx4KjOkauDf9c+Apx61M= 2024-03-04T16:33:44.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ON/YpS7qx4KjOkauDf9c+Apx61M=', 'Content-Length': '0'} 2024-03-04T16:33:44.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d082ec61c4d1e0b2-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.187 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.187 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.187 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.188 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.188 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.188 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.188 DEBUG:boto:Data: 2024-03-04T16:33:44.188 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.188 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.188 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.188 DEBUG:boto:Params: {} 2024-03-04T16:33:44.188 DEBUG:boto:Token: None 2024-03-04T16:33:44.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ON/YpS7qx4KjOkauDf9c+Apx61M= 2024-03-04T16:33:44.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ON/YpS7qx4KjOkauDf9c+Apx61M=', 'Content-Length': '0'} 2024-03-04T16:33:44.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075b8e48424635b26-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.191 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.191 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.191 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.191 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.191 DEBUG:boto:Method: GET 2024-03-04T16:33:44.192 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.192 DEBUG:boto:Data: 2024-03-04T16:33:44.192 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.192 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.192 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.192 DEBUG:boto:Params: {} 2024-03-04T16:33:44.192 DEBUG:boto:Token: None 2024-03-04T16:33:44.192 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.192 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI= 2024-03-04T16:33:44.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI=', 'Content-Length': '0'} 2024-03-04T16:33:44.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d13b34ad471569c2-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.194 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.195 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.195 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.195 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.195 DEBUG:boto:Method: GET 2024-03-04T16:33:44.195 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.195 DEBUG:boto:Data: 2024-03-04T16:33:44.195 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.195 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.195 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.195 DEBUG:boto:Params: {} 2024-03-04T16:33:44.195 DEBUG:boto:Token: None 2024-03-04T16:33:44.196 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:44.196 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI= 2024-03-04T16:33:44.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0Tsk+4eiBliHKRk5MZ/B+v3wxqI=', 'Content-Length': '0'} 2024-03-04T16:33:44.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b282c50c84c5b0c-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.198 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.198 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.198 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.198 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.198 DEBUG:boto:Method: GET 2024-03-04T16:33:44.198 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.198 DEBUG:boto:Data: 2024-03-04T16:33:44.198 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.198 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.198 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.198 DEBUG:boto:Params: {} 2024-03-04T16:33:44.199 DEBUG:boto:Token: None 2024-03-04T16:33:44.199 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.199 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f9d7d969ae0f21-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.202 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.202 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.202 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.202 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.202 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.203 DEBUG:boto:Method: GET 2024-03-04T16:33:44.203 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.203 DEBUG:boto:Data: 2024-03-04T16:33:44.203 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.203 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.203 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.203 DEBUG:boto:Params: {} 2024-03-04T16:33:44.203 DEBUG:boto:Token: None 2024-03-04T16:33:44.203 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.203 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bbe880fe4b5df82-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.206 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.206 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.207 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.207 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.207 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.207 DEBUG:boto:Data: 2024-03-04T16:33:44.207 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.207 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.207 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.207 DEBUG:boto:Params: {} 2024-03-04T16:33:44.207 DEBUG:boto:Token: None 2024-03-04T16:33:44.207 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.208 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e803094c9ea9a98-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.211 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.211 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.211 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.211 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.211 DEBUG:boto:Data: 2024-03-04T16:33:44.211 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.211 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.211 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.211 DEBUG:boto:Params: {} 2024-03-04T16:33:44.211 DEBUG:boto:Token: None 2024-03-04T16:33:44.212 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.212 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007daa75090d2b7a64-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.215 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.215 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.215 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.215 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.215 DEBUG:boto:Method: GET 2024-03-04T16:33:44.215 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.215 DEBUG:boto:Data: 2024-03-04T16:33:44.215 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.215 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.216 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.216 DEBUG:boto:Params: {} 2024-03-04T16:33:44.216 DEBUG:boto:Token: None 2024-03-04T16:33:44.216 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.216 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072808573dd72ad39-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.264 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.264 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.264 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.265 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.265 DEBUG:boto:Method: GET 2024-03-04T16:33:44.265 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.265 DEBUG:boto:Data: 2024-03-04T16:33:44.265 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.265 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.265 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.265 DEBUG:boto:Params: {} 2024-03-04T16:33:44.265 DEBUG:boto:Token: None 2024-03-04T16:33:44.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000703ac4338364ccbf-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.308 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.309 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.309 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.309 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.309 DEBUG:boto:Method: GET 2024-03-04T16:33:44.309 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.309 DEBUG:boto:Data: 2024-03-04T16:33:44.309 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.309 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.309 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.309 DEBUG:boto:Params: {} 2024-03-04T16:33:44.309 DEBUG:boto:Token: None 2024-03-04T16:33:44.310 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2882fb6abf4aed-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.313 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.313 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.313 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.313 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.313 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.313 DEBUG:boto:Method: GET 2024-03-04T16:33:44.314 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.314 DEBUG:boto:Data: 2024-03-04T16:33:44.314 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.314 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.314 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.314 DEBUG:boto:Params: {} 2024-03-04T16:33:44.314 DEBUG:boto:Token: None 2024-03-04T16:33:44.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f19fb33a90e48863-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.317 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.318 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.318 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.318 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.318 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.318 DEBUG:boto:Method: GET 2024-03-04T16:33:44.318 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.318 DEBUG:boto:Data: 2024-03-04T16:33:44.319 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.319 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.319 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.319 DEBUG:boto:Params: {} 2024-03-04T16:33:44.319 DEBUG:boto:Token: None 2024-03-04T16:33:44.319 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.319 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4= 2024-03-04T16:33:44.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001f225aef947b53b-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.321 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.322 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.322 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.322 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.322 DEBUG:boto:Method: GET 2024-03-04T16:33:44.322 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.322 DEBUG:boto:Data: 2024-03-04T16:33:44.322 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.322 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.322 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.322 DEBUG:boto:Params: {} 2024-03-04T16:33:44.322 DEBUG:boto:Token: None 2024-03-04T16:33:44.323 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4= 2024-03-04T16:33:44.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8653419f6ed176f-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.372 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.372 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.372 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.372 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.373 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.373 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.373 DEBUG:boto:Data: 2024-03-04T16:33:44.373 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.373 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.373 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.373 DEBUG:boto:Params: {} 2024-03-04T16:33:44.373 DEBUG:boto:Token: None 2024-03-04T16:33:44.373 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A6hMMo71FxPPWPilQ1jOQ5pZTPU= 2024-03-04T16:33:44.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A6hMMo71FxPPWPilQ1jOQ5pZTPU=', 'Content-Length': '0'} 2024-03-04T16:33:44.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da94994539664f49-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.376 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.376 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.376 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.376 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.376 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.376 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.376 DEBUG:boto:Data: 2024-03-04T16:33:44.376 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.376 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.377 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.377 DEBUG:boto:Params: {} 2024-03-04T16:33:44.377 DEBUG:boto:Token: None 2024-03-04T16:33:44.377 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.377 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A6hMMo71FxPPWPilQ1jOQ5pZTPU= 2024-03-04T16:33:44.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A6hMMo71FxPPWPilQ1jOQ5pZTPU=', 'Content-Length': '0'} 2024-03-04T16:33:44.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009370163979c83be5-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.379 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.379 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.379 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.380 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.380 DEBUG:boto:Method: GET 2024-03-04T16:33:44.380 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.380 DEBUG:boto:Data: 2024-03-04T16:33:44.380 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.380 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.380 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.380 DEBUG:boto:Params: {} 2024-03-04T16:33:44.380 DEBUG:boto:Token: None 2024-03-04T16:33:44.380 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.381 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4= 2024-03-04T16:33:44.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f98e5b7dfbfbbe89-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.383 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.383 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.383 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.383 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.383 DEBUG:boto:Method: GET 2024-03-04T16:33:44.383 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.384 DEBUG:boto:Data: 2024-03-04T16:33:44.384 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.384 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.384 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.384 DEBUG:boto:Params: {} 2024-03-04T16:33:44.384 DEBUG:boto:Token: None 2024-03-04T16:33:44.384 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:44.384 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4= 2024-03-04T16:33:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vIpf883qWFdzsydM99yNicW/IP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000986ff681db842f7b-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.428 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.429 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.429 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.429 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.429 DEBUG:boto:Method: GET 2024-03-04T16:33:44.429 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.429 DEBUG:boto:Data: 2024-03-04T16:33:44.429 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.429 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.429 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.429 DEBUG:boto:Params: {} 2024-03-04T16:33:44.430 DEBUG:boto:Token: None 2024-03-04T16:33:44.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000036111cda62b9304-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.433 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.433 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.433 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.433 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.434 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.434 DEBUG:boto:Method: GET 2024-03-04T16:33:44.434 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.434 DEBUG:boto:Data: 2024-03-04T16:33:44.434 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.434 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.434 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.434 DEBUG:boto:Params: {} 2024-03-04T16:33:44.434 DEBUG:boto:Token: None 2024-03-04T16:33:44.434 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.435 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f351829d0f1226-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.437 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.437 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.438 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.438 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.438 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.438 DEBUG:boto:Data: 2024-03-04T16:33:44.438 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.438 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.438 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.438 DEBUG:boto:Params: {} 2024-03-04T16:33:44.438 DEBUG:boto:Token: None 2024-03-04T16:33:44.438 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.439 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c312b192f448e61-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.442 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.442 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.442 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.442 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.442 DEBUG:boto:Data: 2024-03-04T16:33:44.442 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.442 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.442 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.443 DEBUG:boto:Params: {} 2024-03-04T16:33:44.443 DEBUG:boto:Token: None 2024-03-04T16:33:44.443 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062e8de99413a50ec-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.446 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.446 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.446 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.447 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.447 DEBUG:boto:Method: GET 2024-03-04T16:33:44.447 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.447 DEBUG:boto:Data: 2024-03-04T16:33:44.447 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.447 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.447 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.447 DEBUG:boto:Params: {} 2024-03-04T16:33:44.447 DEBUG:boto:Token: None 2024-03-04T16:33:44.447 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.448 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aabbd46777ff18bc-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.492 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.492 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.492 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.492 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.493 DEBUG:boto:Method: GET 2024-03-04T16:33:44.493 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.493 DEBUG:boto:Data: 2024-03-04T16:33:44.493 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.493 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.493 DEBUG:boto:Params: {} 2024-03-04T16:33:44.493 DEBUG:boto:Token: None 2024-03-04T16:33:44.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b34fab0f1992e473-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.536 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.536 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.536 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.537 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.537 DEBUG:boto:Method: GET 2024-03-04T16:33:44.537 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.537 DEBUG:boto:Data: 2024-03-04T16:33:44.537 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.537 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.537 DEBUG:boto:Params: {} 2024-03-04T16:33:44.537 DEBUG:boto:Token: None 2024-03-04T16:33:44.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f853f97c40f4a458-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.540 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.541 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.541 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.541 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.541 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.541 DEBUG:boto:Method: GET 2024-03-04T16:33:44.541 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.541 DEBUG:boto:Data: 2024-03-04T16:33:44.541 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.541 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.542 DEBUG:boto:Params: {} 2024-03-04T16:33:44.542 DEBUG:boto:Token: None 2024-03-04T16:33:44.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2489cee115715c-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.545 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.545 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.545 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.545 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.545 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.545 DEBUG:boto:Method: GET 2024-03-04T16:33:44.546 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.546 DEBUG:boto:Data: 2024-03-04T16:33:44.546 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.546 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.546 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.546 DEBUG:boto:Params: {} 2024-03-04T16:33:44.546 DEBUG:boto:Token: None 2024-03-04T16:33:44.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA= 2024-03-04T16:33:44.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA=', 'Content-Length': '0'} 2024-03-04T16:33:44.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d779b90d12e49244-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.548 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.549 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.549 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.549 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.549 DEBUG:boto:Method: GET 2024-03-04T16:33:44.549 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.549 DEBUG:boto:Data: 2024-03-04T16:33:44.549 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.549 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.549 DEBUG:boto:Params: {} 2024-03-04T16:33:44.549 DEBUG:boto:Token: None 2024-03-04T16:33:44.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA= 2024-03-04T16:33:44.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA=', 'Content-Length': '0'} 2024-03-04T16:33:44.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcf0faafadf3f9fe-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.596 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.596 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.596 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.597 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.597 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.597 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.597 DEBUG:boto:Data: 2024-03-04T16:33:44.597 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.597 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.597 DEBUG:boto:Params: {} 2024-03-04T16:33:44.597 DEBUG:boto:Token: None 2024-03-04T16:33:44.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tjP+kUf3VhgtibV1yx4ZqN51Or0= 2024-03-04T16:33:44.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tjP+kUf3VhgtibV1yx4ZqN51Or0=', 'Content-Length': '0'} 2024-03-04T16:33:44.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090b277e3a6f27239-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.600 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.600 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.600 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.600 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.600 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.600 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.600 DEBUG:boto:Data: 2024-03-04T16:33:44.600 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.600 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.600 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.600 DEBUG:boto:Params: {} 2024-03-04T16:33:44.601 DEBUG:boto:Token: None 2024-03-04T16:33:44.601 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tjP+kUf3VhgtibV1yx4ZqN51Or0= 2024-03-04T16:33:44.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tjP+kUf3VhgtibV1yx4ZqN51Or0=', 'Content-Length': '0'} 2024-03-04T16:33:44.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da0998f6f0100c89-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.603 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.603 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.603 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.603 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.603 DEBUG:boto:Method: GET 2024-03-04T16:33:44.603 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.604 DEBUG:boto:Data: 2024-03-04T16:33:44.604 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.604 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.604 DEBUG:boto:Params: {} 2024-03-04T16:33:44.604 DEBUG:boto:Token: None 2024-03-04T16:33:44.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.604 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA= 2024-03-04T16:33:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA=', 'Content-Length': '0'} 2024-03-04T16:33:44.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002822a22ad2ff4c3f-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.606 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.606 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.606 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.607 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.607 DEBUG:boto:Method: GET 2024-03-04T16:33:44.607 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.607 DEBUG:boto:Data: 2024-03-04T16:33:44.607 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.607 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.607 DEBUG:boto:Params: {} 2024-03-04T16:33:44.607 DEBUG:boto:Token: None 2024-03-04T16:33:44.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:44.607 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA= 2024-03-04T16:33:44.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S552RGB9GtCbzcAdixMTMvi68HA=', 'Content-Length': '0'} 2024-03-04T16:33:44.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4b8ea6f5d982ae-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.656 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.656 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.656 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.656 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.657 DEBUG:boto:Method: GET 2024-03-04T16:33:44.657 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.657 DEBUG:boto:Data: 2024-03-04T16:33:44.657 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.657 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.657 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.657 DEBUG:boto:Params: {} 2024-03-04T16:33:44.657 DEBUG:boto:Token: None 2024-03-04T16:33:44.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.657 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f9e158968dd589-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.660 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.661 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.661 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.661 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.661 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.661 DEBUG:boto:Method: GET 2024-03-04T16:33:44.661 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.661 DEBUG:boto:Data: 2024-03-04T16:33:44.661 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.661 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.662 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.662 DEBUG:boto:Params: {} 2024-03-04T16:33:44.662 DEBUG:boto:Token: None 2024-03-04T16:33:44.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c4a884f51617b5-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.665 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.665 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.665 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.665 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.665 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.666 DEBUG:boto:Data: 2024-03-04T16:33:44.666 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.666 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.666 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.666 DEBUG:boto:Params: {} 2024-03-04T16:33:44.666 DEBUG:boto:Token: None 2024-03-04T16:33:44.666 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.666 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298370e0c2c23045-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.670 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.670 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.670 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.670 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.670 DEBUG:boto:Data: 2024-03-04T16:33:44.670 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.670 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.670 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.670 DEBUG:boto:Params: {} 2024-03-04T16:33:44.670 DEBUG:boto:Token: None 2024-03-04T16:33:44.671 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.671 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009614803b5f8de92f-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.675 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.675 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.675 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.675 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.675 DEBUG:boto:Method: GET 2024-03-04T16:33:44.675 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.675 DEBUG:boto:Data: 2024-03-04T16:33:44.676 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.676 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.676 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.676 DEBUG:boto:Params: {} 2024-03-04T16:33:44.676 DEBUG:boto:Token: None 2024-03-04T16:33:44.676 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.677 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b03cb045ffc18832-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.720 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.720 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.720 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.720 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.721 DEBUG:boto:Method: GET 2024-03-04T16:33:44.721 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.721 DEBUG:boto:Data: 2024-03-04T16:33:44.723 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.723 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.723 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.724 DEBUG:boto:Params: {} 2024-03-04T16:33:44.724 DEBUG:boto:Token: None 2024-03-04T16:33:44.724 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.724 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c6d9bc9ae6879bc-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.772 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.772 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.772 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.773 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.773 DEBUG:boto:Method: GET 2024-03-04T16:33:44.773 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.773 DEBUG:boto:Data: 2024-03-04T16:33:44.773 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.773 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.773 DEBUG:boto:Params: {} 2024-03-04T16:33:44.773 DEBUG:boto:Token: None 2024-03-04T16:33:44.774 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219c3656a6f69e46-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.777 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.777 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.778 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.778 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.778 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.778 DEBUG:boto:Method: GET 2024-03-04T16:33:44.778 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.778 DEBUG:boto:Data: 2024-03-04T16:33:44.778 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.778 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.778 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.778 DEBUG:boto:Params: {} 2024-03-04T16:33:44.779 DEBUG:boto:Token: None 2024-03-04T16:33:44.779 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.779 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca3df178288d880-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.782 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.783 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.783 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.783 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.783 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.783 DEBUG:boto:Method: GET 2024-03-04T16:33:44.783 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.783 DEBUG:boto:Data: 2024-03-04T16:33:44.783 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.783 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.783 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.784 DEBUG:boto:Params: {} 2024-03-04T16:33:44.784 DEBUG:boto:Token: None 2024-03-04T16:33:44.784 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.784 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A= 2024-03-04T16:33:44.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A=', 'Content-Length': '0'} 2024-03-04T16:33:44.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cd78e43841e0605-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.786 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.786 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.787 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.787 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.787 DEBUG:boto:Method: GET 2024-03-04T16:33:44.787 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.787 DEBUG:boto:Data: 2024-03-04T16:33:44.787 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.787 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.787 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.787 DEBUG:boto:Params: {} 2024-03-04T16:33:44.787 DEBUG:boto:Token: None 2024-03-04T16:33:44.787 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.788 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A= 2024-03-04T16:33:44.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A=', 'Content-Length': '0'} 2024-03-04T16:33:44.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfe103d05c0241e6-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.790 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.790 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.790 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.790 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.790 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.790 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.790 DEBUG:boto:Data: 2024-03-04T16:33:44.790 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.791 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.791 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.791 DEBUG:boto:Params: {} 2024-03-04T16:33:44.791 DEBUG:boto:Token: None 2024-03-04T16:33:44.791 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+u+F2myLKl8G0TpS0PIY7SnN/Vg= 2024-03-04T16:33:44.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+u+F2myLKl8G0TpS0PIY7SnN/Vg=', 'Content-Length': '0'} 2024-03-04T16:33:44.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009508b03a06b3032d-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.793 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.793 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.794 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.794 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.794 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.794 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.794 DEBUG:boto:Data: 2024-03-04T16:33:44.794 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.794 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.794 DEBUG:boto:Params: {} 2024-03-04T16:33:44.794 DEBUG:boto:Token: None 2024-03-04T16:33:44.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+u+F2myLKl8G0TpS0PIY7SnN/Vg= 2024-03-04T16:33:44.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+u+F2myLKl8G0TpS0PIY7SnN/Vg=', 'Content-Length': '0'} 2024-03-04T16:33:44.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000782666c0e1b03a46-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.797 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.797 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.797 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.797 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.797 DEBUG:boto:Method: GET 2024-03-04T16:33:44.797 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.797 DEBUG:boto:Data: 2024-03-04T16:33:44.798 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.798 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.798 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.798 DEBUG:boto:Params: {} 2024-03-04T16:33:44.798 DEBUG:boto:Token: None 2024-03-04T16:33:44.798 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.798 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A= 2024-03-04T16:33:44.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A=', 'Content-Length': '0'} 2024-03-04T16:33:44.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8e9fdd1b0889f91-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.800 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.800 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.800 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.800 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.800 DEBUG:boto:Method: GET 2024-03-04T16:33:44.801 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.801 DEBUG:boto:Data: 2024-03-04T16:33:44.801 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.801 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.801 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.801 DEBUG:boto:Params: {} 2024-03-04T16:33:44.801 DEBUG:boto:Token: None 2024-03-04T16:33:44.801 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:44.802 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A= 2024-03-04T16:33:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hJOZOpjzXr7rhvGlmM+Y3dlrS3A=', 'Content-Length': '0'} 2024-03-04T16:33:44.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d180070512b6a2a-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.844 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.844 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.845 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.845 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.845 DEBUG:boto:Method: GET 2024-03-04T16:33:44.845 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.845 DEBUG:boto:Data: 2024-03-04T16:33:44.845 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.845 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.845 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.845 DEBUG:boto:Params: {} 2024-03-04T16:33:44.846 DEBUG:boto:Token: None 2024-03-04T16:33:44.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aec846eb02426e3-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.849 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.849 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.849 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.849 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.850 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.850 DEBUG:boto:Method: GET 2024-03-04T16:33:44.850 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.850 DEBUG:boto:Data: 2024-03-04T16:33:44.850 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.850 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.850 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.850 DEBUG:boto:Params: {} 2024-03-04T16:33:44.850 DEBUG:boto:Token: None 2024-03-04T16:33:44.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1ba300d868086a-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.853 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.854 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.854 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.854 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.854 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.854 DEBUG:boto:Data: 2024-03-04T16:33:44.854 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.854 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.854 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.854 DEBUG:boto:Params: {} 2024-03-04T16:33:44.854 DEBUG:boto:Token: None 2024-03-04T16:33:44.855 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053b5c8f238618a0f-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.858 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.858 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.858 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.858 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:44.858 DEBUG:boto:Data: 2024-03-04T16:33:44.858 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.858 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.858 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.858 DEBUG:boto:Params: {} 2024-03-04T16:33:44.859 DEBUG:boto:Token: None 2024-03-04T16:33:44.859 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj 2024-03-04T16:33:44.859 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU= 2024-03-04T16:33:44.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:386xW6wXEFVRb8GyRVNgRv0SNQU=', 'Content-Length': '0'} 2024-03-04T16:33:44.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6c884c6d1eaad24-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.866 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.866 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.866 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.866 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.866 DEBUG:boto:Method: GET 2024-03-04T16:33:44.866 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.866 DEBUG:boto:Data: 2024-03-04T16:33:44.867 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.867 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.867 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.867 DEBUG:boto:Params: {} 2024-03-04T16:33:44.867 DEBUG:boto:Token: None 2024-03-04T16:33:44.867 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.867 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000235f51f4807e75c1-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.912 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.912 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.912 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.913 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.913 DEBUG:boto:Method: GET 2024-03-04T16:33:44.913 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.913 DEBUG:boto:Data: 2024-03-04T16:33:44.913 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.913 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.913 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.913 DEBUG:boto:Params: {} 2024-03-04T16:33:44.913 DEBUG:boto:Token: None 2024-03-04T16:33:44.914 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:44.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8= 2024-03-04T16:33:44.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JDwdae3VuXrS/IY0/pjgi7es5Q8=', 'Content-Length': '0'} 2024-03-04T16:33:44.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000247655fb1837017-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.960 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.960 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.960 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.960 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.961 DEBUG:boto:Method: GET 2024-03-04T16:33:44.961 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.961 DEBUG:boto:Data: 2024-03-04T16:33:44.961 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.961 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.961 DEBUG:boto:Params: {} 2024-03-04T16:33:44.961 DEBUG:boto:Token: None 2024-03-04T16:33:44.961 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.961 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f870f0339dc63b-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.965 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.965 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.965 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.965 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.965 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.965 DEBUG:boto:Method: GET 2024-03-04T16:33:44.966 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.966 DEBUG:boto:Data: 2024-03-04T16:33:44.966 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.966 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.966 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.966 DEBUG:boto:Params: {} 2024-03-04T16:33:44.966 DEBUG:boto:Token: None 2024-03-04T16:33:44.967 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.967 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15ea22732589dfd-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.970 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.970 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.970 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.970 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.971 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.971 DEBUG:boto:Method: GET 2024-03-04T16:33:44.971 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.972 DEBUG:boto:Data: 2024-03-04T16:33:44.972 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.972 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.972 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.972 DEBUG:boto:Params: {} 2024-03-04T16:33:44.972 DEBUG:boto:Token: None 2024-03-04T16:33:44.972 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.973 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao= 2024-03-04T16:33:44.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao=', 'Content-Length': '0'} 2024-03-04T16:33:44.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c23dcb31c4e1550-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.975 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.975 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.975 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.975 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.975 DEBUG:boto:Method: GET 2024-03-04T16:33:44.975 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.975 DEBUG:boto:Data: 2024-03-04T16:33:44.976 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.976 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.976 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.976 DEBUG:boto:Params: {} 2024-03-04T16:33:44.976 DEBUG:boto:Token: None 2024-03-04T16:33:44.976 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.976 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao= 2024-03-04T16:33:44.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao=', 'Content-Length': '0'} 2024-03-04T16:33:44.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f093ccf92cb1a25-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.978 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.979 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.979 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.979 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.979 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.979 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.979 DEBUG:boto:Data: 2024-03-04T16:33:44.979 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.979 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.979 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.979 DEBUG:boto:Params: {} 2024-03-04T16:33:44.980 DEBUG:boto:Token: None 2024-03-04T16:33:44.980 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.980 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x79D0gKU5+YQbHoQZweoW2xqtP4= 2024-03-04T16:33:44.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x79D0gKU5+YQbHoQZweoW2xqtP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abe4034f5d9d60fc-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.982 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.982 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.982 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.982 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.983 DEBUG:boto:Method: HEAD 2024-03-04T16:33:44.983 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.983 DEBUG:boto:Data: 2024-03-04T16:33:44.983 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.983 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.983 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.983 DEBUG:boto:Params: {} 2024-03-04T16:33:44.983 DEBUG:boto:Token: None 2024-03-04T16:33:44.983 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.984 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:x79D0gKU5+YQbHoQZweoW2xqtP4= 2024-03-04T16:33:44.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:x79D0gKU5+YQbHoQZweoW2xqtP4=', 'Content-Length': '0'} 2024-03-04T16:33:44.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7bdc2c51788dbc9-0065e5f7e8-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:44.986 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.986 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.986 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.986 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.986 DEBUG:boto:Method: GET 2024-03-04T16:33:44.987 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.987 DEBUG:boto:Data: 2024-03-04T16:33:44.987 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.987 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.987 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.987 DEBUG:boto:Params: {} 2024-03-04T16:33:44.987 DEBUG:boto:Token: None 2024-03-04T16:33:44.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao= 2024-03-04T16:33:44.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao=', 'Content-Length': '0'} 2024-03-04T16:33:44.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000624a4ef4efe39d3c-0065e5f7e8-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:44 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.989 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:44.990 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:44.990 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.990 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.990 DEBUG:boto:Method: GET 2024-03-04T16:33:44.990 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.990 DEBUG:boto:Data: 2024-03-04T16:33:44.990 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.990 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.990 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.990 DEBUG:boto:Params: {} 2024-03-04T16:33:44.991 DEBUG:boto:Token: None 2024-03-04T16:33:44.991 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:44.991 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao= 2024-03-04T16:33:44.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AoQ+Gi+i25wU1pXsiwoYFtg0+Ao=', 'Content-Length': '0'} 2024-03-04T16:33:44.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8a46e0083d4d83a-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.993 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.993 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.994 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.994 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.994 DEBUG:boto:Method: GET 2024-03-04T16:33:44.994 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.994 DEBUG:boto:Data: 2024-03-04T16:33:44.994 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.994 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:44.994 DEBUG:boto:Port: 8001 2024-03-04T16:33:44.994 DEBUG:boto:Params: {} 2024-03-04T16:33:44.994 DEBUG:boto:Token: None 2024-03-04T16:33:44.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.995 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:44.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f293b762358df348-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:44.997 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:44.997 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:44.998 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:44.998 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.998 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:44.998 DEBUG:boto:Method: GET 2024-03-04T16:33:44.998 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:44.998 DEBUG:boto:Data: 2024-03-04T16:33:44.998 DEBUG:boto:Headers: {} 2024-03-04T16:33:44.998 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:44.998 DEBUG:boto:Port: 8000 2024-03-04T16:33:44.998 DEBUG:boto:Params: {} 2024-03-04T16:33:44.998 DEBUG:boto:Token: None 2024-03-04T16:33:44.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:44 GMT /sbnvju-22/ 2024-03-04T16:33:44.999 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI= 2024-03-04T16:33:44.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:44 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIpnhFVl8LHuqaF7PB5UC9Qw9ZI=', 'Content-Length': '0'} 2024-03-04T16:33:45.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b2a9be7281af21f-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.048 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.049 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.049 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.049 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.049 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.049 DEBUG:boto:Data: 2024-03-04T16:33:45.049 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.049 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.049 DEBUG:boto:Params: {} 2024-03-04T16:33:45.050 DEBUG:boto:Token: None 2024-03-04T16:33:45.050 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cbb8bd8a6c78b62-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.054 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.054 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.054 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.054 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.054 DEBUG:boto:Data: 2024-03-04T16:33:45.054 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.054 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.054 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.055 DEBUG:boto:Params: {} 2024-03-04T16:33:45.055 DEBUG:boto:Token: None 2024-03-04T16:33:45.055 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.055 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f8592c416b76cb2-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.059 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.059 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.059 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.059 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.059 DEBUG:boto:Method: GET 2024-03-04T16:33:45.059 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.059 DEBUG:boto:Data: 2024-03-04T16:33:45.060 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.060 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.060 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.060 DEBUG:boto:Params: {} 2024-03-04T16:33:45.060 DEBUG:boto:Token: None 2024-03-04T16:33:45.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.060 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c2c75cae523596a-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.062 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.063 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.063 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.063 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.063 DEBUG:boto:Method: GET 2024-03-04T16:33:45.063 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.063 DEBUG:boto:Data: 2024-03-04T16:33:45.063 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.063 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.063 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.064 DEBUG:boto:Params: {} 2024-03-04T16:33:45.064 DEBUG:boto:Token: None 2024-03-04T16:33:45.064 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.064 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025fc4333914f71a5-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.108 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.108 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.108 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.109 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.109 DEBUG:boto:Method: GET 2024-03-04T16:33:45.109 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.109 DEBUG:boto:Data: 2024-03-04T16:33:45.109 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.109 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.109 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.109 DEBUG:boto:Params: {} 2024-03-04T16:33:45.109 DEBUG:boto:Token: None 2024-03-04T16:33:45.109 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab434f38f319e94-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.113 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.114 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.114 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.114 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.114 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.114 DEBUG:boto:Method: GET 2024-03-04T16:33:45.114 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.114 DEBUG:boto:Data: 2024-03-04T16:33:45.114 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.114 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.114 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.114 DEBUG:boto:Params: {} 2024-03-04T16:33:45.115 DEBUG:boto:Token: None 2024-03-04T16:33:45.115 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.115 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034abf171364198a2-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.160 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.161 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.161 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.161 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.161 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.161 DEBUG:boto:Method: GET 2024-03-04T16:33:45.161 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.161 DEBUG:boto:Data: 2024-03-04T16:33:45.161 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.161 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.162 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.162 DEBUG:boto:Params: {} 2024-03-04T16:33:45.162 DEBUG:boto:Token: None 2024-03-04T16:33:45.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.162 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y= 2024-03-04T16:33:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y=', 'Content-Length': '0'} 2024-03-04T16:33:45.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001acb57c5ccb2d35-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.165 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.165 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.165 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.165 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.165 DEBUG:boto:Method: GET 2024-03-04T16:33:45.165 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.165 DEBUG:boto:Data: 2024-03-04T16:33:45.165 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.165 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.165 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.166 DEBUG:boto:Params: {} 2024-03-04T16:33:45.166 DEBUG:boto:Token: None 2024-03-04T16:33:45.166 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.166 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y= 2024-03-04T16:33:45.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y=', 'Content-Length': '0'} 2024-03-04T16:33:45.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a307087d39631f8b-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.168 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.169 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.169 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.169 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.169 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.169 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.169 DEBUG:boto:Data: 2024-03-04T16:33:45.169 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.169 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.169 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.169 DEBUG:boto:Params: {} 2024-03-04T16:33:45.170 DEBUG:boto:Token: None 2024-03-04T16:33:45.170 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V7JMZZClSuMr06CvN2o0RGp8/ts= 2024-03-04T16:33:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V7JMZZClSuMr06CvN2o0RGp8/ts=', 'Content-Length': '0'} 2024-03-04T16:33:45.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c63cd808702e9e2-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.172 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.172 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.172 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.172 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.172 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.172 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.172 DEBUG:boto:Data: 2024-03-04T16:33:45.173 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.173 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.173 DEBUG:boto:Params: {} 2024-03-04T16:33:45.173 DEBUG:boto:Token: None 2024-03-04T16:33:45.173 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V7JMZZClSuMr06CvN2o0RGp8/ts= 2024-03-04T16:33:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V7JMZZClSuMr06CvN2o0RGp8/ts=', 'Content-Length': '0'} 2024-03-04T16:33:45.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000517fcc8f749fea2c-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.176 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.176 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.176 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.176 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.176 DEBUG:boto:Method: GET 2024-03-04T16:33:45.176 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.176 DEBUG:boto:Data: 2024-03-04T16:33:45.177 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.177 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.177 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.177 DEBUG:boto:Params: {} 2024-03-04T16:33:45.177 DEBUG:boto:Token: None 2024-03-04T16:33:45.177 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.177 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y= 2024-03-04T16:33:45.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y=', 'Content-Length': '0'} 2024-03-04T16:33:45.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2b916c2802973f2-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.224 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.224 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.224 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.225 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.225 DEBUG:boto:Method: GET 2024-03-04T16:33:45.225 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.225 DEBUG:boto:Data: 2024-03-04T16:33:45.225 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.225 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.225 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.225 DEBUG:boto:Params: {} 2024-03-04T16:33:45.225 DEBUG:boto:Token: None 2024-03-04T16:33:45.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:45.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y= 2024-03-04T16:33:45.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aopOyKkMXeDjX2JUgNrzBPx502Y=', 'Content-Length': '0'} 2024-03-04T16:33:45.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fc3147281db2479-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.268 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.268 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.268 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.269 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.269 DEBUG:boto:Method: GET 2024-03-04T16:33:45.269 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.269 DEBUG:boto:Data: 2024-03-04T16:33:45.269 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.269 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.269 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.269 DEBUG:boto:Params: {} 2024-03-04T16:33:45.269 DEBUG:boto:Token: None 2024-03-04T16:33:45.269 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098c01702734f5672-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.273 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.273 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.273 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.273 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.273 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.274 DEBUG:boto:Method: GET 2024-03-04T16:33:45.274 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.274 DEBUG:boto:Data: 2024-03-04T16:33:45.274 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.274 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.274 DEBUG:boto:Params: {} 2024-03-04T16:33:45.274 DEBUG:boto:Token: None 2024-03-04T16:33:45.275 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63ee7035e6859a3-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.324 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.325 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.325 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.325 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.325 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.325 DEBUG:boto:Data: 2024-03-04T16:33:45.325 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.325 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.325 DEBUG:boto:Params: {} 2024-03-04T16:33:45.325 DEBUG:boto:Token: None 2024-03-04T16:33:45.326 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000455689e0d3930a1c-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.330 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.330 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.330 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.330 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.330 DEBUG:boto:Data: 2024-03-04T16:33:45.330 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.330 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.330 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.330 DEBUG:boto:Params: {} 2024-03-04T16:33:45.331 DEBUG:boto:Token: None 2024-03-04T16:33:45.331 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.331 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000614b057e8380fe1b-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.336 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.337 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.337 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.337 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.337 DEBUG:boto:Method: GET 2024-03-04T16:33:45.337 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.337 DEBUG:boto:Data: 2024-03-04T16:33:45.337 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.337 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.337 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.337 DEBUG:boto:Params: {} 2024-03-04T16:33:45.337 DEBUG:boto:Token: None 2024-03-04T16:33:45.337 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000762cba4e8c45c7a6-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.340 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.340 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.340 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.340 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.340 DEBUG:boto:Method: GET 2024-03-04T16:33:45.340 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.340 DEBUG:boto:Data: 2024-03-04T16:33:45.340 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.340 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.341 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.341 DEBUG:boto:Params: {} 2024-03-04T16:33:45.341 DEBUG:boto:Token: None 2024-03-04T16:33:45.341 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.341 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b09152af6768367-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.384 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.384 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.384 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.384 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.385 DEBUG:boto:Method: GET 2024-03-04T16:33:45.385 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.385 DEBUG:boto:Data: 2024-03-04T16:33:45.385 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.385 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.385 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.385 DEBUG:boto:Params: {} 2024-03-04T16:33:45.385 DEBUG:boto:Token: None 2024-03-04T16:33:45.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.386 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d16be716c43721-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.388 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.389 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.389 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.389 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.389 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.389 DEBUG:boto:Method: GET 2024-03-04T16:33:45.389 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.389 DEBUG:boto:Data: 2024-03-04T16:33:45.389 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.389 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.390 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.390 DEBUG:boto:Params: {} 2024-03-04T16:33:45.390 DEBUG:boto:Token: None 2024-03-04T16:33:45.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7be127160653990-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.436 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.437 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.437 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.437 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.437 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.437 DEBUG:boto:Method: GET 2024-03-04T16:33:45.437 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.437 DEBUG:boto:Data: 2024-03-04T16:33:45.437 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.437 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.437 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.438 DEBUG:boto:Params: {} 2024-03-04T16:33:45.438 DEBUG:boto:Token: None 2024-03-04T16:33:45.438 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.438 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY= 2024-03-04T16:33:45.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY=', 'Content-Length': '0'} 2024-03-04T16:33:45.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d101d48d5a4f4047-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.440 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.440 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.441 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.441 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.441 DEBUG:boto:Method: GET 2024-03-04T16:33:45.441 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.441 DEBUG:boto:Data: 2024-03-04T16:33:45.441 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.441 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.441 DEBUG:boto:Params: {} 2024-03-04T16:33:45.441 DEBUG:boto:Token: None 2024-03-04T16:33:45.442 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.442 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY= 2024-03-04T16:33:45.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY=', 'Content-Length': '0'} 2024-03-04T16:33:45.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000816c7249e7d6f60a-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.488 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.488 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.488 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.488 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.489 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.489 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.489 DEBUG:boto:Data: 2024-03-04T16:33:45.489 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.489 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.489 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.489 DEBUG:boto:Params: {} 2024-03-04T16:33:45.489 DEBUG:boto:Token: None 2024-03-04T16:33:45.489 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.489 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sr/i308KSLmoEOJ0p4UGb/4XWjw= 2024-03-04T16:33:45.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sr/i308KSLmoEOJ0p4UGb/4XWjw=', 'Content-Length': '0'} 2024-03-04T16:33:45.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5ed628b100e64fa-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.492 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.492 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.492 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.492 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.492 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.492 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.492 DEBUG:boto:Data: 2024-03-04T16:33:45.492 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.492 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.493 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.493 DEBUG:boto:Params: {} 2024-03-04T16:33:45.493 DEBUG:boto:Token: None 2024-03-04T16:33:45.493 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sr/i308KSLmoEOJ0p4UGb/4XWjw= 2024-03-04T16:33:45.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sr/i308KSLmoEOJ0p4UGb/4XWjw=', 'Content-Length': '0'} 2024-03-04T16:33:45.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec5578900d6a4e2d-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.495 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.495 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.495 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.495 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.495 DEBUG:boto:Method: GET 2024-03-04T16:33:45.495 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.495 DEBUG:boto:Data: 2024-03-04T16:33:45.496 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.496 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.496 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.496 DEBUG:boto:Params: {} 2024-03-04T16:33:45.496 DEBUG:boto:Token: None 2024-03-04T16:33:45.496 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.496 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY= 2024-03-04T16:33:45.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY=', 'Content-Length': '0'} 2024-03-04T16:33:45.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c25f94bf98c255b8-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.498 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.498 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.498 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.498 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.499 DEBUG:boto:Method: GET 2024-03-04T16:33:45.499 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.499 DEBUG:boto:Data: 2024-03-04T16:33:45.499 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.499 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.499 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.499 DEBUG:boto:Params: {} 2024-03-04T16:33:45.499 DEBUG:boto:Token: None 2024-03-04T16:33:45.499 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:45.499 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY= 2024-03-04T16:33:45.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mBFyQ4zqQWh6+VON2QxmfINNfrY=', 'Content-Length': '0'} 2024-03-04T16:33:45.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df0811e49980d075-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.544 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.544 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.544 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.545 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.545 DEBUG:boto:Method: GET 2024-03-04T16:33:45.545 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.545 DEBUG:boto:Data: 2024-03-04T16:33:45.545 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.545 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.545 DEBUG:boto:Params: {} 2024-03-04T16:33:45.545 DEBUG:boto:Token: None 2024-03-04T16:33:45.545 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7168830d436e818-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.549 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.549 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.549 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.549 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.549 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.549 DEBUG:boto:Method: GET 2024-03-04T16:33:45.550 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.550 DEBUG:boto:Data: 2024-03-04T16:33:45.550 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.550 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.550 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.550 DEBUG:boto:Params: {} 2024-03-04T16:33:45.550 DEBUG:boto:Token: None 2024-03-04T16:33:45.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c405e065aeac0a-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.596 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.597 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.597 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.597 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.597 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.597 DEBUG:boto:Data: 2024-03-04T16:33:45.597 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.597 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.597 DEBUG:boto:Params: {} 2024-03-04T16:33:45.598 DEBUG:boto:Token: None 2024-03-04T16:33:45.598 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3f75250749fb225-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.601 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.601 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.601 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.601 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.602 DEBUG:boto:Data: 2024-03-04T16:33:45.602 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.602 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.602 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.602 DEBUG:boto:Params: {} 2024-03-04T16:33:45.602 DEBUG:boto:Token: None 2024-03-04T16:33:45.602 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.602 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a57bb92025a580f-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.606 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.606 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.606 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.606 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.606 DEBUG:boto:Method: GET 2024-03-04T16:33:45.606 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.606 DEBUG:boto:Data: 2024-03-04T16:33:45.607 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.607 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.607 DEBUG:boto:Params: {} 2024-03-04T16:33:45.607 DEBUG:boto:Token: None 2024-03-04T16:33:45.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.607 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d6f1e317ec8179-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.609 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.610 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.610 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.610 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.610 DEBUG:boto:Method: GET 2024-03-04T16:33:45.610 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.610 DEBUG:boto:Data: 2024-03-04T16:33:45.610 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.610 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.610 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.610 DEBUG:boto:Params: {} 2024-03-04T16:33:45.610 DEBUG:boto:Token: None 2024-03-04T16:33:45.611 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.611 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bb63918e1e4fc5c-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.660 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.660 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.660 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.661 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.661 DEBUG:boto:Method: GET 2024-03-04T16:33:45.661 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.661 DEBUG:boto:Data: 2024-03-04T16:33:45.661 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.661 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.661 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.661 DEBUG:boto:Params: {} 2024-03-04T16:33:45.661 DEBUG:boto:Token: None 2024-03-04T16:33:45.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081628b91d3ea1cd0-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.664 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.665 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.665 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.665 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.665 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.665 DEBUG:boto:Method: GET 2024-03-04T16:33:45.665 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.666 DEBUG:boto:Data: 2024-03-04T16:33:45.666 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.666 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.666 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.666 DEBUG:boto:Params: {} 2024-03-04T16:33:45.666 DEBUG:boto:Token: None 2024-03-04T16:33:45.666 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.666 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c445eb928619a2-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.712 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.713 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.713 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.713 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.713 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.713 DEBUG:boto:Method: GET 2024-03-04T16:33:45.713 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.713 DEBUG:boto:Data: 2024-03-04T16:33:45.714 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.714 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.714 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.714 DEBUG:boto:Params: {} 2024-03-04T16:33:45.714 DEBUG:boto:Token: None 2024-03-04T16:33:45.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE= 2024-03-04T16:33:45.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE=', 'Content-Length': '0'} 2024-03-04T16:33:45.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000062ab4a322c7e817-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.717 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.717 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.717 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.717 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.717 DEBUG:boto:Method: GET 2024-03-04T16:33:45.718 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.718 DEBUG:boto:Data: 2024-03-04T16:33:45.718 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.718 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.718 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.718 DEBUG:boto:Params: {} 2024-03-04T16:33:45.718 DEBUG:boto:Token: None 2024-03-04T16:33:45.718 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.718 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE= 2024-03-04T16:33:45.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE=', 'Content-Length': '0'} 2024-03-04T16:33:45.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdb34db7969d758e-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.721 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.721 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.721 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.721 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.721 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.721 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.721 DEBUG:boto:Data: 2024-03-04T16:33:45.722 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.722 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.722 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.722 DEBUG:boto:Params: {} 2024-03-04T16:33:45.722 DEBUG:boto:Token: None 2024-03-04T16:33:45.722 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TRqCeFRLJmfGLDsHfnAVk79WSvY= 2024-03-04T16:33:45.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TRqCeFRLJmfGLDsHfnAVk79WSvY=', 'Content-Length': '0'} 2024-03-04T16:33:45.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d12ffcb20647964-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.724 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.724 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.724 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.725 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.725 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.725 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.725 DEBUG:boto:Data: 2024-03-04T16:33:45.725 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.725 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.725 DEBUG:boto:Params: {} 2024-03-04T16:33:45.725 DEBUG:boto:Token: None 2024-03-04T16:33:45.725 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TRqCeFRLJmfGLDsHfnAVk79WSvY= 2024-03-04T16:33:45.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TRqCeFRLJmfGLDsHfnAVk79WSvY=', 'Content-Length': '0'} 2024-03-04T16:33:45.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072a82848b13a7c4e-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.727 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.727 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.728 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.728 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.728 DEBUG:boto:Method: GET 2024-03-04T16:33:45.728 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.728 DEBUG:boto:Data: 2024-03-04T16:33:45.728 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.728 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.728 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.728 DEBUG:boto:Params: {} 2024-03-04T16:33:45.728 DEBUG:boto:Token: None 2024-03-04T16:33:45.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.729 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE= 2024-03-04T16:33:45.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE=', 'Content-Length': '0'} 2024-03-04T16:33:45.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079a0b466e4c69fc6-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.772 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.772 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.772 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.772 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.773 DEBUG:boto:Method: GET 2024-03-04T16:33:45.773 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.773 DEBUG:boto:Data: 2024-03-04T16:33:45.773 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.773 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.773 DEBUG:boto:Params: {} 2024-03-04T16:33:45.773 DEBUG:boto:Token: None 2024-03-04T16:33:45.773 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:45.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE= 2024-03-04T16:33:45.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uw12uPrWwSVikeIz5/uYYRtKgwE=', 'Content-Length': '0'} 2024-03-04T16:33:45.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fca600cf7a6a3fc4-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.776 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.776 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.776 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.776 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.776 DEBUG:boto:Method: GET 2024-03-04T16:33:45.776 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.777 DEBUG:boto:Data: 2024-03-04T16:33:45.777 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.777 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.777 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.777 DEBUG:boto:Params: {} 2024-03-04T16:33:45.777 DEBUG:boto:Token: None 2024-03-04T16:33:45.777 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.777 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7715ebf0a44f028-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.780 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.781 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.781 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.781 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.781 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.781 DEBUG:boto:Method: GET 2024-03-04T16:33:45.781 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.781 DEBUG:boto:Data: 2024-03-04T16:33:45.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.782 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.782 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.782 DEBUG:boto:Params: {} 2024-03-04T16:33:45.782 DEBUG:boto:Token: None 2024-03-04T16:33:45.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201bf81c980cf1f8-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.785 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.785 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.785 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.785 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.786 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.786 DEBUG:boto:Data: 2024-03-04T16:33:45.786 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.786 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.786 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.786 DEBUG:boto:Params: {} 2024-03-04T16:33:45.786 DEBUG:boto:Token: None 2024-03-04T16:33:45.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000844ef66ca6ef141e-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.790 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.790 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.790 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.790 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:45.790 DEBUG:boto:Data: 2024-03-04T16:33:45.790 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.790 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.790 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.790 DEBUG:boto:Params: {} 2024-03-04T16:33:45.791 DEBUG:boto:Token: None 2024-03-04T16:33:45.791 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj 2024-03-04T16:33:45.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag= 2024-03-04T16:33:45.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OqHsUztw6CR7GvTJe3lcxfUWlag=', 'Content-Length': '0'} 2024-03-04T16:33:45.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005683883c50bdb653-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.794 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.794 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.795 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.795 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.795 DEBUG:boto:Method: GET 2024-03-04T16:33:45.795 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.795 DEBUG:boto:Data: 2024-03-04T16:33:45.795 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.795 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.795 DEBUG:boto:Params: {} 2024-03-04T16:33:45.795 DEBUG:boto:Token: None 2024-03-04T16:33:45.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.796 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021dd79089ef4fbfd-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.840 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.840 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.840 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.841 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.841 DEBUG:boto:Method: GET 2024-03-04T16:33:45.841 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.841 DEBUG:boto:Data: 2024-03-04T16:33:45.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.841 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.841 DEBUG:boto:Params: {} 2024-03-04T16:33:45.841 DEBUG:boto:Token: None 2024-03-04T16:33:45.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:45.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA= 2024-03-04T16:33:45.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GWwJgl0NkWdaYj69sc/GoO+UUgA=', 'Content-Length': '0'} 2024-03-04T16:33:45.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7aeb715e033a5e6-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.888 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.888 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.888 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.888 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.889 DEBUG:boto:Method: GET 2024-03-04T16:33:45.889 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.889 DEBUG:boto:Data: 2024-03-04T16:33:45.889 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.889 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.889 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.889 DEBUG:boto:Params: {} 2024-03-04T16:33:45.889 DEBUG:boto:Token: None 2024-03-04T16:33:45.889 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74738bcaca28af1-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.892 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.893 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.893 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.893 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.893 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.893 DEBUG:boto:Method: GET 2024-03-04T16:33:45.893 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.893 DEBUG:boto:Data: 2024-03-04T16:33:45.893 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.894 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.894 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.894 DEBUG:boto:Params: {} 2024-03-04T16:33:45.894 DEBUG:boto:Token: None 2024-03-04T16:33:45.894 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.894 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beedde0669625269-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.897 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.897 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.898 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.898 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.898 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.898 DEBUG:boto:Method: GET 2024-03-04T16:33:45.898 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.898 DEBUG:boto:Data: 2024-03-04T16:33:45.898 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.898 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.898 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.898 DEBUG:boto:Params: {} 2024-03-04T16:33:45.898 DEBUG:boto:Token: None 2024-03-04T16:33:45.899 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.899 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4= 2024-03-04T16:33:45.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4=', 'Content-Length': '0'} 2024-03-04T16:33:45.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eac97e4ef9edac7-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.901 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.901 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.901 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.901 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.901 DEBUG:boto:Method: GET 2024-03-04T16:33:45.901 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.901 DEBUG:boto:Data: 2024-03-04T16:33:45.901 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.901 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.902 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.902 DEBUG:boto:Params: {} 2024-03-04T16:33:45.902 DEBUG:boto:Token: None 2024-03-04T16:33:45.902 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.902 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4= 2024-03-04T16:33:45.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4=', 'Content-Length': '0'} 2024-03-04T16:33:45.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046f5dc1bd2b501d4-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.904 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.904 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.904 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.905 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.905 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.905 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.905 DEBUG:boto:Data: 2024-03-04T16:33:45.905 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.906 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.906 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.906 DEBUG:boto:Params: {} 2024-03-04T16:33:45.906 DEBUG:boto:Token: None 2024-03-04T16:33:45.906 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.906 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7lNKowFMJKdi0X+NBJC6icnficU= 2024-03-04T16:33:45.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7lNKowFMJKdi0X+NBJC6icnficU=', 'Content-Length': '0'} 2024-03-04T16:33:45.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056261f3b6901d8a9-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.908 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.908 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.908 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.908 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.908 DEBUG:boto:Method: HEAD 2024-03-04T16:33:45.909 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.909 DEBUG:boto:Data: 2024-03-04T16:33:45.909 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.909 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.909 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.909 DEBUG:boto:Params: {} 2024-03-04T16:33:45.909 DEBUG:boto:Token: None 2024-03-04T16:33:45.909 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.909 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7lNKowFMJKdi0X+NBJC6icnficU= 2024-03-04T16:33:45.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7lNKowFMJKdi0X+NBJC6icnficU=', 'Content-Length': '0'} 2024-03-04T16:33:45.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000937c9e33e6fcb9db-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:45.911 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.911 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.912 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.912 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.912 DEBUG:boto:Method: GET 2024-03-04T16:33:45.912 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.912 DEBUG:boto:Data: 2024-03-04T16:33:45.912 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.912 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.912 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.912 DEBUG:boto:Params: {} 2024-03-04T16:33:45.912 DEBUG:boto:Token: None 2024-03-04T16:33:45.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4= 2024-03-04T16:33:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4=', 'Content-Length': '0'} 2024-03-04T16:33:45.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e32d64c9e5b67d3-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.915 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:45.915 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:45.915 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.915 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.915 DEBUG:boto:Method: GET 2024-03-04T16:33:45.915 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.915 DEBUG:boto:Data: 2024-03-04T16:33:45.916 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.916 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.916 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.916 DEBUG:boto:Params: {} 2024-03-04T16:33:45.916 DEBUG:boto:Token: None 2024-03-04T16:33:45.916 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:45.916 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4= 2024-03-04T16:33:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jR5YzmGJ3O9Uc+gkJAptezYT0F4=', 'Content-Length': '0'} 2024-03-04T16:33:45.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b92be7957e353cd7-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.964 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.964 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.964 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.965 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.965 DEBUG:boto:Method: GET 2024-03-04T16:33:45.965 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.965 DEBUG:boto:Data: 2024-03-04T16:33:45.965 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.965 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:45.965 DEBUG:boto:Port: 8001 2024-03-04T16:33:45.965 DEBUG:boto:Params: {} 2024-03-04T16:33:45.966 DEBUG:boto:Token: None 2024-03-04T16:33:45.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec286268ee0bf1b9-0065e5f7e9-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:45.969 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:45.969 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:45.969 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:45.969 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.969 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:45.970 DEBUG:boto:Method: GET 2024-03-04T16:33:45.970 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:45.970 DEBUG:boto:Data: 2024-03-04T16:33:45.970 DEBUG:boto:Headers: {} 2024-03-04T16:33:45.970 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:45.970 DEBUG:boto:Port: 8000 2024-03-04T16:33:45.970 DEBUG:boto:Params: {} 2024-03-04T16:33:45.970 DEBUG:boto:Token: None 2024-03-04T16:33:45.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:45 GMT /sbnvju-22/ 2024-03-04T16:33:45.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8= 2024-03-04T16:33:45.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xb8rd7YyxnvE1qPMCRx/Fl7Scw8=', 'Content-Length': '0'} 2024-03-04T16:33:45.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000680f9fd61350273-0065e5f7e9-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.020 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.021 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.021 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.021 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.021 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.021 DEBUG:boto:Data: 2024-03-04T16:33:46.021 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.021 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.021 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.021 DEBUG:boto:Params: {} 2024-03-04T16:33:46.022 DEBUG:boto:Token: None 2024-03-04T16:33:46.022 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036ffcfb44d3c2363-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.026 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.026 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.026 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.026 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.026 DEBUG:boto:Data: 2024-03-04T16:33:46.026 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.026 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.026 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.026 DEBUG:boto:Params: {} 2024-03-04T16:33:46.027 DEBUG:boto:Token: None 2024-03-04T16:33:46.027 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.027 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbfac588a9960934-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.031 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.031 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.031 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.031 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.031 DEBUG:boto:Method: GET 2024-03-04T16:33:46.031 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.031 DEBUG:boto:Data: 2024-03-04T16:33:46.031 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.031 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.031 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.031 DEBUG:boto:Params: {} 2024-03-04T16:33:46.032 DEBUG:boto:Token: None 2024-03-04T16:33:46.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.032 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e355418a5738b46-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.076 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.076 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.076 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.076 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.077 DEBUG:boto:Method: GET 2024-03-04T16:33:46.077 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.077 DEBUG:boto:Data: 2024-03-04T16:33:46.077 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.077 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.077 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.077 DEBUG:boto:Params: {} 2024-03-04T16:33:46.077 DEBUG:boto:Token: None 2024-03-04T16:33:46.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6de40ca1768e127-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.120 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.120 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.121 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.121 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.121 DEBUG:boto:Method: GET 2024-03-04T16:33:46.121 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.121 DEBUG:boto:Data: 2024-03-04T16:33:46.121 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.121 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.121 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.122 DEBUG:boto:Params: {} 2024-03-04T16:33:46.122 DEBUG:boto:Token: None 2024-03-04T16:33:46.122 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9ebc3bbe3633f6-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.125 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.126 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.126 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.126 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.126 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.126 DEBUG:boto:Method: GET 2024-03-04T16:33:46.126 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.127 DEBUG:boto:Data: 2024-03-04T16:33:46.127 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.127 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.127 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.127 DEBUG:boto:Params: {} 2024-03-04T16:33:46.127 DEBUG:boto:Token: None 2024-03-04T16:33:46.127 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.127 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f09fb1794969be-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.172 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.173 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.173 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.173 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.173 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.173 DEBUG:boto:Method: GET 2024-03-04T16:33:46.173 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.174 DEBUG:boto:Data: 2024-03-04T16:33:46.174 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.174 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.174 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.174 DEBUG:boto:Params: {} 2024-03-04T16:33:46.174 DEBUG:boto:Token: None 2024-03-04T16:33:46.174 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.174 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc= 2024-03-04T16:33:46.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc=', 'Content-Length': '0'} 2024-03-04T16:33:46.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d7fb3cbdfebbce0-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.179 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.179 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.179 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.179 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.179 DEBUG:boto:Method: GET 2024-03-04T16:33:46.179 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.179 DEBUG:boto:Data: 2024-03-04T16:33:46.179 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.179 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.179 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.179 DEBUG:boto:Params: {} 2024-03-04T16:33:46.180 DEBUG:boto:Token: None 2024-03-04T16:33:46.180 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.180 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc= 2024-03-04T16:33:46.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc=', 'Content-Length': '0'} 2024-03-04T16:33:46.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f21a81230bc20796-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.183 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.183 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.183 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.183 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.183 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.183 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.183 DEBUG:boto:Data: 2024-03-04T16:33:46.183 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.183 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.183 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.183 DEBUG:boto:Params: {} 2024-03-04T16:33:46.184 DEBUG:boto:Token: None 2024-03-04T16:33:46.184 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.184 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O0re2PrxQey0w1Q1T8aCWtkIz84= 2024-03-04T16:33:46.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O0re2PrxQey0w1Q1T8aCWtkIz84=', 'Content-Length': '0'} 2024-03-04T16:33:46.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e689a8d39532c63e-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.186 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.186 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.186 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.186 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.186 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.186 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.186 DEBUG:boto:Data: 2024-03-04T16:33:46.187 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.187 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.187 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.188 DEBUG:boto:Params: {} 2024-03-04T16:33:46.188 DEBUG:boto:Token: None 2024-03-04T16:33:46.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.188 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O0re2PrxQey0w1Q1T8aCWtkIz84= 2024-03-04T16:33:46.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O0re2PrxQey0w1Q1T8aCWtkIz84=', 'Content-Length': '0'} 2024-03-04T16:33:46.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020183f2c2f566aef-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.190 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.190 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.191 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.191 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.191 DEBUG:boto:Method: GET 2024-03-04T16:33:46.191 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.191 DEBUG:boto:Data: 2024-03-04T16:33:46.191 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.191 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.191 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.191 DEBUG:boto:Params: {} 2024-03-04T16:33:46.191 DEBUG:boto:Token: None 2024-03-04T16:33:46.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.192 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc= 2024-03-04T16:33:46.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc=', 'Content-Length': '0'} 2024-03-04T16:33:46.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a93418e294a2c7-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.194 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.194 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.194 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.194 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.194 DEBUG:boto:Method: GET 2024-03-04T16:33:46.194 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.194 DEBUG:boto:Data: 2024-03-04T16:33:46.194 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.195 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.195 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.195 DEBUG:boto:Params: {} 2024-03-04T16:33:46.195 DEBUG:boto:Token: None 2024-03-04T16:33:46.195 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:46.195 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc= 2024-03-04T16:33:46.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZN++UAyARl2ne8MxoH2HhvfdFtc=', 'Content-Length': '0'} 2024-03-04T16:33:46.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3f2a59b8ee62d66-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.240 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.240 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.240 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.241 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.241 DEBUG:boto:Method: GET 2024-03-04T16:33:46.241 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.241 DEBUG:boto:Data: 2024-03-04T16:33:46.241 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.241 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.241 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.241 DEBUG:boto:Params: {} 2024-03-04T16:33:46.241 DEBUG:boto:Token: None 2024-03-04T16:33:46.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003430ebe99edd66d3-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.244 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.245 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.245 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.245 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.245 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.245 DEBUG:boto:Method: GET 2024-03-04T16:33:46.245 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.245 DEBUG:boto:Data: 2024-03-04T16:33:46.246 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.246 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.246 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.246 DEBUG:boto:Params: {} 2024-03-04T16:33:46.246 DEBUG:boto:Token: None 2024-03-04T16:33:46.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000455c202231160ac-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.249 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.250 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.250 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.250 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.250 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.250 DEBUG:boto:Data: 2024-03-04T16:33:46.250 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.250 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.250 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.250 DEBUG:boto:Params: {} 2024-03-04T16:33:46.251 DEBUG:boto:Token: None 2024-03-04T16:33:46.251 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.251 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0803ffd45d44c13-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.254 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.254 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.254 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.255 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.255 DEBUG:boto:Data: 2024-03-04T16:33:46.255 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.255 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.255 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.255 DEBUG:boto:Params: {} 2024-03-04T16:33:46.255 DEBUG:boto:Token: None 2024-03-04T16:33:46.255 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.255 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006672e801e9f1c7e8-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.259 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.259 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.259 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.259 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.259 DEBUG:boto:Method: GET 2024-03-04T16:33:46.259 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.259 DEBUG:boto:Data: 2024-03-04T16:33:46.259 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.260 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.260 DEBUG:boto:Params: {} 2024-03-04T16:33:46.260 DEBUG:boto:Token: None 2024-03-04T16:33:46.260 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.260 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000447f4db67192271c-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.304 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.304 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.304 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.304 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.305 DEBUG:boto:Method: GET 2024-03-04T16:33:46.305 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.305 DEBUG:boto:Data: 2024-03-04T16:33:46.305 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.305 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.305 DEBUG:boto:Params: {} 2024-03-04T16:33:46.305 DEBUG:boto:Token: None 2024-03-04T16:33:46.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a26f112dc9e7f374-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.348 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.348 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.348 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.348 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.349 DEBUG:boto:Method: GET 2024-03-04T16:33:46.349 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.349 DEBUG:boto:Data: 2024-03-04T16:33:46.349 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.349 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.349 DEBUG:boto:Params: {} 2024-03-04T16:33:46.349 DEBUG:boto:Token: None 2024-03-04T16:33:46.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db3b0b9296072a6-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.352 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.353 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.353 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.353 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.353 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.353 DEBUG:boto:Method: GET 2024-03-04T16:33:46.353 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.353 DEBUG:boto:Data: 2024-03-04T16:33:46.353 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.354 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.354 DEBUG:boto:Params: {} 2024-03-04T16:33:46.354 DEBUG:boto:Token: None 2024-03-04T16:33:46.354 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe1f6f6b8b3e5e1-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.360 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.361 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.361 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.361 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.361 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.361 DEBUG:boto:Method: GET 2024-03-04T16:33:46.361 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.361 DEBUG:boto:Data: 2024-03-04T16:33:46.361 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.361 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.361 DEBUG:boto:Params: {} 2024-03-04T16:33:46.362 DEBUG:boto:Token: None 2024-03-04T16:33:46.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0= 2024-03-04T16:33:46.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0=', 'Content-Length': '0'} 2024-03-04T16:33:46.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ab2560a599591c0-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.404 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.404 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.404 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.404 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.405 DEBUG:boto:Method: GET 2024-03-04T16:33:46.405 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.405 DEBUG:boto:Data: 2024-03-04T16:33:46.405 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.405 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.405 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.405 DEBUG:boto:Params: {} 2024-03-04T16:33:46.405 DEBUG:boto:Token: None 2024-03-04T16:33:46.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0= 2024-03-04T16:33:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0=', 'Content-Length': '0'} 2024-03-04T16:33:46.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a280f768f7e783b-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.408 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.408 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.408 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.408 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.408 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.409 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.409 DEBUG:boto:Data: 2024-03-04T16:33:46.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.409 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.409 DEBUG:boto:Params: {} 2024-03-04T16:33:46.409 DEBUG:boto:Token: None 2024-03-04T16:33:46.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Lu0j2vTDX927INCHwMy++pSr4IE= 2024-03-04T16:33:46.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Lu0j2vTDX927INCHwMy++pSr4IE=', 'Content-Length': '0'} 2024-03-04T16:33:46.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ac76f6bcbddde6-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.412 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.412 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.412 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.412 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.412 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.412 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.412 DEBUG:boto:Data: 2024-03-04T16:33:46.412 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.412 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.412 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.412 DEBUG:boto:Params: {} 2024-03-04T16:33:46.413 DEBUG:boto:Token: None 2024-03-04T16:33:46.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Lu0j2vTDX927INCHwMy++pSr4IE= 2024-03-04T16:33:46.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Lu0j2vTDX927INCHwMy++pSr4IE=', 'Content-Length': '0'} 2024-03-04T16:33:46.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005171a38db18c0989-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.415 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.415 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.415 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.415 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.415 DEBUG:boto:Method: GET 2024-03-04T16:33:46.415 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.415 DEBUG:boto:Data: 2024-03-04T16:33:46.415 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.416 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.416 DEBUG:boto:Params: {} 2024-03-04T16:33:46.416 DEBUG:boto:Token: None 2024-03-04T16:33:46.416 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.416 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0= 2024-03-04T16:33:46.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0=', 'Content-Length': '0'} 2024-03-04T16:33:46.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000608d07e413cf9f2f-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.464 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.464 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.464 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.465 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.465 DEBUG:boto:Method: GET 2024-03-04T16:33:46.465 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.465 DEBUG:boto:Data: 2024-03-04T16:33:46.465 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.465 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.465 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.465 DEBUG:boto:Params: {} 2024-03-04T16:33:46.465 DEBUG:boto:Token: None 2024-03-04T16:33:46.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:46.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0= 2024-03-04T16:33:46.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V2lQnyIvPo2lAzPX7JC6xYpFfE0=', 'Content-Length': '0'} 2024-03-04T16:33:46.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4ed1b1da16dbc64-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.468 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.468 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.468 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.468 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.468 DEBUG:boto:Method: GET 2024-03-04T16:33:46.468 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.469 DEBUG:boto:Data: 2024-03-04T16:33:46.469 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.469 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.469 DEBUG:boto:Params: {} 2024-03-04T16:33:46.469 DEBUG:boto:Token: None 2024-03-04T16:33:46.469 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.469 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000de14fd94efdb142-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.516 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.517 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.517 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.517 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.517 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.517 DEBUG:boto:Method: GET 2024-03-04T16:33:46.517 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.517 DEBUG:boto:Data: 2024-03-04T16:33:46.517 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.518 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.518 DEBUG:boto:Params: {} 2024-03-04T16:33:46.518 DEBUG:boto:Token: None 2024-03-04T16:33:46.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac324dad8467e78-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.522 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.522 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.522 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.522 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.522 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.522 DEBUG:boto:Data: 2024-03-04T16:33:46.522 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.523 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.523 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.523 DEBUG:boto:Params: {} 2024-03-04T16:33:46.523 DEBUG:boto:Token: None 2024-03-04T16:33:46.523 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.523 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005ba8b019d0e3a75-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.527 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.527 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.527 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.527 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.527 DEBUG:boto:Data: 2024-03-04T16:33:46.527 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.527 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.528 DEBUG:boto:Params: {} 2024-03-04T16:33:46.528 DEBUG:boto:Token: None 2024-03-04T16:33:46.528 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035912433417b8ba1-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.532 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.532 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.532 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.532 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.532 DEBUG:boto:Method: GET 2024-03-04T16:33:46.532 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.532 DEBUG:boto:Data: 2024-03-04T16:33:46.533 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.533 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.533 DEBUG:boto:Params: {} 2024-03-04T16:33:46.533 DEBUG:boto:Token: None 2024-03-04T16:33:46.533 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4e282989251867e-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.580 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.580 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.580 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.580 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.581 DEBUG:boto:Method: GET 2024-03-04T16:33:46.581 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.581 DEBUG:boto:Data: 2024-03-04T16:33:46.581 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.581 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.581 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.581 DEBUG:boto:Params: {} 2024-03-04T16:33:46.581 DEBUG:boto:Token: None 2024-03-04T16:33:46.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064500797f306a913-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.624 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.624 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.624 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.625 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.625 DEBUG:boto:Method: GET 2024-03-04T16:33:46.625 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.625 DEBUG:boto:Data: 2024-03-04T16:33:46.625 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.625 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.625 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.625 DEBUG:boto:Params: {} 2024-03-04T16:33:46.625 DEBUG:boto:Token: None 2024-03-04T16:33:46.625 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d758cd82b0ee68b-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.672 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.673 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.673 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.673 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.673 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.673 DEBUG:boto:Method: GET 2024-03-04T16:33:46.673 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.673 DEBUG:boto:Data: 2024-03-04T16:33:46.673 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.673 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.674 DEBUG:boto:Params: {} 2024-03-04T16:33:46.674 DEBUG:boto:Token: None 2024-03-04T16:33:46.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cae985aba6eac6b2-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.677 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.678 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.678 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.678 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.678 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.678 DEBUG:boto:Method: GET 2024-03-04T16:33:46.678 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.678 DEBUG:boto:Data: 2024-03-04T16:33:46.678 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.678 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.679 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.679 DEBUG:boto:Params: {} 2024-03-04T16:33:46.679 DEBUG:boto:Token: None 2024-03-04T16:33:46.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.679 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA= 2024-03-04T16:33:46.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA=', 'Content-Length': '0'} 2024-03-04T16:33:46.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9bdb559a4a5f3ab-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.724 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.724 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.724 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.724 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.725 DEBUG:boto:Method: GET 2024-03-04T16:33:46.725 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.725 DEBUG:boto:Data: 2024-03-04T16:33:46.725 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.725 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.725 DEBUG:boto:Params: {} 2024-03-04T16:33:46.725 DEBUG:boto:Token: None 2024-03-04T16:33:46.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA= 2024-03-04T16:33:46.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA=', 'Content-Length': '0'} 2024-03-04T16:33:46.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c1009b6719ce957-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.728 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.728 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.728 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.728 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.728 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.728 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.728 DEBUG:boto:Data: 2024-03-04T16:33:46.729 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.729 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.729 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.729 DEBUG:boto:Params: {} 2024-03-04T16:33:46.729 DEBUG:boto:Token: None 2024-03-04T16:33:46.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.729 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W9WS1Hh4U3XOlXNnNVLFes4n1pE= 2024-03-04T16:33:46.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W9WS1Hh4U3XOlXNnNVLFes4n1pE=', 'Content-Length': '0'} 2024-03-04T16:33:46.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e6e6cabed8820a4-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.731 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.731 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.731 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.731 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.732 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.732 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.732 DEBUG:boto:Data: 2024-03-04T16:33:46.732 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.732 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.732 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.732 DEBUG:boto:Params: {} 2024-03-04T16:33:46.732 DEBUG:boto:Token: None 2024-03-04T16:33:46.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.732 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W9WS1Hh4U3XOlXNnNVLFes4n1pE= 2024-03-04T16:33:46.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W9WS1Hh4U3XOlXNnNVLFes4n1pE=', 'Content-Length': '0'} 2024-03-04T16:33:46.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ba0ac8c34c0314a-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.735 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.735 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.735 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.735 DEBUG:boto:Method: GET 2024-03-04T16:33:46.735 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.735 DEBUG:boto:Data: 2024-03-04T16:33:46.735 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.735 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.735 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.736 DEBUG:boto:Params: {} 2024-03-04T16:33:46.736 DEBUG:boto:Token: None 2024-03-04T16:33:46.736 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA= 2024-03-04T16:33:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA=', 'Content-Length': '0'} 2024-03-04T16:33:46.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ad38928c056607d-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.784 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.784 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.785 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.785 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.785 DEBUG:boto:Method: GET 2024-03-04T16:33:46.785 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.785 DEBUG:boto:Data: 2024-03-04T16:33:46.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.785 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.785 DEBUG:boto:Params: {} 2024-03-04T16:33:46.785 DEBUG:boto:Token: None 2024-03-04T16:33:46.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:46.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA= 2024-03-04T16:33:46.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VBC2C0lK9TXXo6z2UmFqPuuT1dA=', 'Content-Length': '0'} 2024-03-04T16:33:46.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f93a3ca357e1038a-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.788 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.788 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.788 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.788 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.788 DEBUG:boto:Method: GET 2024-03-04T16:33:46.789 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.789 DEBUG:boto:Data: 2024-03-04T16:33:46.789 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.789 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.789 DEBUG:boto:Params: {} 2024-03-04T16:33:46.789 DEBUG:boto:Token: None 2024-03-04T16:33:46.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccebe116f0613855-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.836 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.837 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.837 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.837 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.837 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.837 DEBUG:boto:Method: GET 2024-03-04T16:33:46.837 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.837 DEBUG:boto:Data: 2024-03-04T16:33:46.837 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.837 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.838 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.838 DEBUG:boto:Params: {} 2024-03-04T16:33:46.838 DEBUG:boto:Token: None 2024-03-04T16:33:46.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000378fb8b31b0fb907-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.841 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.841 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.842 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.842 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.842 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.842 DEBUG:boto:Data: 2024-03-04T16:33:46.842 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.842 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.842 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.842 DEBUG:boto:Params: {} 2024-03-04T16:33:46.842 DEBUG:boto:Token: None 2024-03-04T16:33:46.843 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.843 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007294f1ecd6dbf348-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.846 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.846 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.847 DEBUG:boto:Method: HEAD 2024-03-04T16:33:46.847 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:46.847 DEBUG:boto:Data: 2024-03-04T16:33:46.847 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.847 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.847 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.847 DEBUG:boto:Params: {} 2024-03-04T16:33:46.847 DEBUG:boto:Token: None 2024-03-04T16:33:46.847 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj 2024-03-04T16:33:46.847 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU= 2024-03-04T16:33:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u4j8SlBYCb5H9KG7ifSflGJWSlU=', 'Content-Length': '0'} 2024-03-04T16:33:46.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b614e74354652bf8-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.851 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.852 DEBUG:boto:Method: GET 2024-03-04T16:33:46.852 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.852 DEBUG:boto:Data: 2024-03-04T16:33:46.852 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.852 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.852 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.852 DEBUG:boto:Params: {} 2024-03-04T16:33:46.852 DEBUG:boto:Token: None 2024-03-04T16:33:46.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048352965938e3880-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.896 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.896 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.896 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.896 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.897 DEBUG:boto:Method: GET 2024-03-04T16:33:46.897 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.897 DEBUG:boto:Data: 2024-03-04T16:33:46.897 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.897 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.897 DEBUG:boto:Params: {} 2024-03-04T16:33:46.897 DEBUG:boto:Token: None 2024-03-04T16:33:46.898 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:46.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0= 2024-03-04T16:33:46.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jPgnIOYMmMDcUPo20ErsaVWRbJ0=', 'Content-Length': '0'} 2024-03-04T16:33:46.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdbc6028bdab96a9-0065e5f7ea-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.940 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.940 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.940 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.940 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.941 DEBUG:boto:Method: GET 2024-03-04T16:33:46.941 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.941 DEBUG:boto:Data: 2024-03-04T16:33:46.941 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.941 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.941 DEBUG:boto:Params: {} 2024-03-04T16:33:46.941 DEBUG:boto:Token: None 2024-03-04T16:33:46.941 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024823ff2b4603c57-0065e5f7ea-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.992 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.993 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:46.993 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:46.993 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.993 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:46.993 DEBUG:boto:Method: GET 2024-03-04T16:33:46.993 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:46.993 DEBUG:boto:Data: 2024-03-04T16:33:46.994 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.994 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:46.994 DEBUG:boto:Port: 8000 2024-03-04T16:33:46.994 DEBUG:boto:Params: {} 2024-03-04T16:33:46.994 DEBUG:boto:Token: None 2024-03-04T16:33:46.994 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/ 2024-03-04T16:33:46.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY= 2024-03-04T16:33:46.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p3NqV3u3K1k2Xq97yAUWg8X9TYY=', 'Content-Length': '0'} 2024-03-04T16:33:46.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000741463a5fce47f4b-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:46.998 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:46.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:46.998 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:46.998 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:46.998 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:46.998 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:46.999 DEBUG:boto:Method: GET 2024-03-04T16:33:46.999 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:46.999 DEBUG:boto:Data: 2024-03-04T16:33:46.999 DEBUG:boto:Headers: {} 2024-03-04T16:33:46.999 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:46.999 DEBUG:boto:Port: 8001 2024-03-04T16:33:46.999 DEBUG:boto:Params: {} 2024-03-04T16:33:46.999 DEBUG:boto:Token: None 2024-03-04T16:33:46.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:46 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.000 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:r3PtaVcMvLwDTnBui1lCKC3DnHM= 2024-03-04T16:33:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:r3PtaVcMvLwDTnBui1lCKC3DnHM=', 'Content-Length': '0'} 2024-03-04T16:33:47.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004bf06259f5d3ddb-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.044 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.044 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.044 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.044 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.045 DEBUG:boto:Method: GET 2024-03-04T16:33:47.045 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.045 DEBUG:boto:Data: 2024-03-04T16:33:47.045 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.045 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.045 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.045 DEBUG:boto:Params: {} 2024-03-04T16:33:47.045 DEBUG:boto:Token: None 2024-03-04T16:33:47.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4= 2024-03-04T16:33:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4=', 'Content-Length': '0'} 2024-03-04T16:33:47.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000103a9ec6b031cb2c-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.048 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.048 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.048 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.048 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.048 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.049 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.049 DEBUG:boto:Data: 2024-03-04T16:33:47.049 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.049 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.049 DEBUG:boto:Params: {} 2024-03-04T16:33:47.049 DEBUG:boto:Token: None 2024-03-04T16:33:47.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE0suNCGQqlN/m+gyLoYqfRImTA= 2024-03-04T16:33:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE0suNCGQqlN/m+gyLoYqfRImTA=', 'Content-Length': '0'} 2024-03-04T16:33:47.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000077b1fabaacc3365-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.052 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.052 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.052 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.052 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.052 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.052 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.052 DEBUG:boto:Data: 2024-03-04T16:33:47.052 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.052 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.053 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.053 DEBUG:boto:Params: {} 2024-03-04T16:33:47.053 DEBUG:boto:Token: None 2024-03-04T16:33:47.053 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rE0suNCGQqlN/m+gyLoYqfRImTA= 2024-03-04T16:33:47.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rE0suNCGQqlN/m+gyLoYqfRImTA=', 'Content-Length': '0'} 2024-03-04T16:33:47.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cf4e9cb126fcb36-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.055 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.056 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.056 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.056 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.056 DEBUG:boto:Method: GET 2024-03-04T16:33:47.056 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.056 DEBUG:boto:Data: 2024-03-04T16:33:47.056 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.056 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.057 DEBUG:boto:Params: {} 2024-03-04T16:33:47.057 DEBUG:boto:Token: None 2024-03-04T16:33:47.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.057 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4= 2024-03-04T16:33:47.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4=', 'Content-Length': '0'} 2024-03-04T16:33:47.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d29a9e9bfff41cc-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.100 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.100 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.100 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.101 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.101 DEBUG:boto:Method: GET 2024-03-04T16:33:47.101 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.101 DEBUG:boto:Data: 2024-03-04T16:33:47.101 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.101 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.101 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.101 DEBUG:boto:Params: {} 2024-03-04T16:33:47.101 DEBUG:boto:Token: None 2024-03-04T16:33:47.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:47.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4= 2024-03-04T16:33:47.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kI2lGqMW/925py6358FZBHHDxp4=', 'Content-Length': '0'} 2024-03-04T16:33:47.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091c49e783fefd965-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.148 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.148 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.149 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.149 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.149 DEBUG:boto:Method: GET 2024-03-04T16:33:47.149 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.149 DEBUG:boto:Data: 2024-03-04T16:33:47.149 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.149 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.149 DEBUG:boto:Params: {} 2024-03-04T16:33:47.150 DEBUG:boto:Token: None 2024-03-04T16:33:47.150 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e4c0ba80bf4b410-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.200 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.201 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.201 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.201 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.201 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.201 DEBUG:boto:Method: GET 2024-03-04T16:33:47.201 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.201 DEBUG:boto:Data: 2024-03-04T16:33:47.202 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.202 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.202 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.202 DEBUG:boto:Params: {} 2024-03-04T16:33:47.202 DEBUG:boto:Token: None 2024-03-04T16:33:47.202 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcfad170f9c94e3-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.248 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.249 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.249 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.249 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.249 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.249 DEBUG:boto:Data: 2024-03-04T16:33:47.249 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.249 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.249 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.249 DEBUG:boto:Params: {} 2024-03-04T16:33:47.249 DEBUG:boto:Token: None 2024-03-04T16:33:47.250 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.250 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041162958279144cb-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.253 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.253 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.254 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.254 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.254 DEBUG:boto:Data: 2024-03-04T16:33:47.254 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.254 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.254 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.254 DEBUG:boto:Params: {} 2024-03-04T16:33:47.254 DEBUG:boto:Token: None 2024-03-04T16:33:47.254 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4cc28009b7f8e77-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.258 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.258 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.258 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.258 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.258 DEBUG:boto:Method: GET 2024-03-04T16:33:47.258 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.258 DEBUG:boto:Data: 2024-03-04T16:33:47.259 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.259 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.259 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.259 DEBUG:boto:Params: {} 2024-03-04T16:33:47.259 DEBUG:boto:Token: None 2024-03-04T16:33:47.259 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.259 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e41d65f8fbc8f71-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.304 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.304 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.304 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.305 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.305 DEBUG:boto:Method: GET 2024-03-04T16:33:47.305 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.305 DEBUG:boto:Data: 2024-03-04T16:33:47.305 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.305 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.305 DEBUG:boto:Params: {} 2024-03-04T16:33:47.305 DEBUG:boto:Token: None 2024-03-04T16:33:47.305 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000928660b31e55ee94-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.348 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.348 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.348 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.349 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.349 DEBUG:boto:Method: GET 2024-03-04T16:33:47.349 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.349 DEBUG:boto:Data: 2024-03-04T16:33:47.349 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.349 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.349 DEBUG:boto:Params: {} 2024-03-04T16:33:47.349 DEBUG:boto:Token: None 2024-03-04T16:33:47.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc0230501780a5c-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.353 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.354 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.354 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.354 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.354 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.354 DEBUG:boto:Method: GET 2024-03-04T16:33:47.354 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.355 DEBUG:boto:Data: 2024-03-04T16:33:47.355 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.355 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.355 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.355 DEBUG:boto:Params: {} 2024-03-04T16:33:47.355 DEBUG:boto:Token: None 2024-03-04T16:33:47.355 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.355 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000671b1513299e827e-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.400 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.401 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.401 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.401 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.402 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.402 DEBUG:boto:Method: GET 2024-03-04T16:33:47.402 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.402 DEBUG:boto:Data: 2024-03-04T16:33:47.402 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.402 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.402 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.402 DEBUG:boto:Params: {} 2024-03-04T16:33:47.402 DEBUG:boto:Token: None 2024-03-04T16:33:47.403 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.403 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s= 2024-03-04T16:33:47.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s=', 'Content-Length': '0'} 2024-03-04T16:33:47.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036a80f154c4d1813-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.452 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.452 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.452 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.453 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.453 DEBUG:boto:Method: GET 2024-03-04T16:33:47.453 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.453 DEBUG:boto:Data: 2024-03-04T16:33:47.453 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.453 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.453 DEBUG:boto:Params: {} 2024-03-04T16:33:47.453 DEBUG:boto:Token: None 2024-03-04T16:33:47.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s= 2024-03-04T16:33:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s=', 'Content-Length': '0'} 2024-03-04T16:33:47.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83c3e1095303307-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.456 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.456 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.456 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.457 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.457 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.457 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.457 DEBUG:boto:Data: 2024-03-04T16:33:47.457 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.457 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.457 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.457 DEBUG:boto:Params: {} 2024-03-04T16:33:47.457 DEBUG:boto:Token: None 2024-03-04T16:33:47.457 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I6Q9m8fl/+2B76GwygqQX0s8WAE= 2024-03-04T16:33:47.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I6Q9m8fl/+2B76GwygqQX0s8WAE=', 'Content-Length': '0'} 2024-03-04T16:33:47.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000025b5532eaffbc29-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.460 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.460 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.460 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.460 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.460 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.460 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.460 DEBUG:boto:Data: 2024-03-04T16:33:47.460 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.461 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.461 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.461 DEBUG:boto:Params: {} 2024-03-04T16:33:47.461 DEBUG:boto:Token: None 2024-03-04T16:33:47.461 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:I6Q9m8fl/+2B76GwygqQX0s8WAE= 2024-03-04T16:33:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I6Q9m8fl/+2B76GwygqQX0s8WAE=', 'Content-Length': '0'} 2024-03-04T16:33:47.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002462d8ce9984ffcf-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.463 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.464 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.464 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.464 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.464 DEBUG:boto:Method: GET 2024-03-04T16:33:47.464 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.464 DEBUG:boto:Data: 2024-03-04T16:33:47.464 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.464 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.464 DEBUG:boto:Params: {} 2024-03-04T16:33:47.464 DEBUG:boto:Token: None 2024-03-04T16:33:47.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s= 2024-03-04T16:33:47.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s=', 'Content-Length': '0'} 2024-03-04T16:33:47.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3e27258bffc12f7-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.467 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.467 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.467 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.467 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.467 DEBUG:boto:Method: GET 2024-03-04T16:33:47.467 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.467 DEBUG:boto:Data: 2024-03-04T16:33:47.468 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.468 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.468 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.468 DEBUG:boto:Params: {} 2024-03-04T16:33:47.468 DEBUG:boto:Token: None 2024-03-04T16:33:47.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:47.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s= 2024-03-04T16:33:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7fXf1NMo1y1Zt6X2INlx66krt2s=', 'Content-Length': '0'} 2024-03-04T16:33:47.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bd5306ca36824b5-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.513 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.513 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.514 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.514 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.514 DEBUG:boto:Method: GET 2024-03-04T16:33:47.515 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.515 DEBUG:boto:Data: 2024-03-04T16:33:47.515 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.515 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.516 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.517 DEBUG:boto:Params: {} 2024-03-04T16:33:47.517 DEBUG:boto:Token: None 2024-03-04T16:33:47.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2aa53f42e55a79-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.520 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.521 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.521 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.521 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.522 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.522 DEBUG:boto:Method: GET 2024-03-04T16:33:47.522 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.522 DEBUG:boto:Data: 2024-03-04T16:33:47.522 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.522 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.522 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.523 DEBUG:boto:Params: {} 2024-03-04T16:33:47.523 DEBUG:boto:Token: None 2024-03-04T16:33:47.523 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.523 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ebba3c2c70b76b9-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.527 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.527 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.527 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.528 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.528 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.528 DEBUG:boto:Data: 2024-03-04T16:33:47.528 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.528 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.528 DEBUG:boto:Params: {} 2024-03-04T16:33:47.528 DEBUG:boto:Token: None 2024-03-04T16:33:47.528 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.528 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eccc3c10f5ca99e-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.532 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.532 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.532 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.532 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.532 DEBUG:boto:Data: 2024-03-04T16:33:47.533 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.533 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.533 DEBUG:boto:Params: {} 2024-03-04T16:33:47.533 DEBUG:boto:Token: None 2024-03-04T16:33:47.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfd7d4a60637cfaa-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.537 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.537 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.537 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.537 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.537 DEBUG:boto:Method: GET 2024-03-04T16:33:47.537 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.537 DEBUG:boto:Data: 2024-03-04T16:33:47.537 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.537 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.537 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.538 DEBUG:boto:Params: {} 2024-03-04T16:33:47.538 DEBUG:boto:Token: None 2024-03-04T16:33:47.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000632778a2dc25b46b-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.588 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.588 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.592 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.592 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.592 DEBUG:boto:Method: GET 2024-03-04T16:33:47.592 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.592 DEBUG:boto:Data: 2024-03-04T16:33:47.592 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.592 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.592 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.592 DEBUG:boto:Params: {} 2024-03-04T16:33:47.593 DEBUG:boto:Token: None 2024-03-04T16:33:47.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.593 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000866358262ed5425f-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.636 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.636 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.636 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.636 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.637 DEBUG:boto:Method: GET 2024-03-04T16:33:47.637 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.637 DEBUG:boto:Data: 2024-03-04T16:33:47.637 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.637 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.637 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.637 DEBUG:boto:Params: {} 2024-03-04T16:33:47.637 DEBUG:boto:Token: None 2024-03-04T16:33:47.637 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009390f48d550d7c4f-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.684 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.685 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.685 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.685 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.685 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.685 DEBUG:boto:Method: GET 2024-03-04T16:33:47.685 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.685 DEBUG:boto:Data: 2024-03-04T16:33:47.685 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.685 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.686 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.686 DEBUG:boto:Params: {} 2024-03-04T16:33:47.686 DEBUG:boto:Token: None 2024-03-04T16:33:47.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be79dd5c6b8322ce-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.689 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.690 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.690 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.690 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.690 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.690 DEBUG:boto:Method: GET 2024-03-04T16:33:47.690 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.691 DEBUG:boto:Data: 2024-03-04T16:33:47.691 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.691 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.691 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.691 DEBUG:boto:Params: {} 2024-03-04T16:33:47.691 DEBUG:boto:Token: None 2024-03-04T16:33:47.691 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.691 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0= 2024-03-04T16:33:47.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0=', 'Content-Length': '0'} 2024-03-04T16:33:47.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ede425838328e8a9-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.693 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.694 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.694 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.694 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.694 DEBUG:boto:Method: GET 2024-03-04T16:33:47.694 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.694 DEBUG:boto:Data: 2024-03-04T16:33:47.694 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.694 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.694 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.694 DEBUG:boto:Params: {} 2024-03-04T16:33:47.695 DEBUG:boto:Token: None 2024-03-04T16:33:47.695 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.695 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0= 2024-03-04T16:33:47.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0=', 'Content-Length': '0'} 2024-03-04T16:33:47.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081540f69e631c131-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.697 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.697 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.697 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.697 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.697 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.697 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.698 DEBUG:boto:Data: 2024-03-04T16:33:47.698 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.698 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.698 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.698 DEBUG:boto:Params: {} 2024-03-04T16:33:47.698 DEBUG:boto:Token: None 2024-03-04T16:33:47.698 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.699 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hZmKTnH8QvCRqRiCzYv/CziUC3s= 2024-03-04T16:33:47.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hZmKTnH8QvCRqRiCzYv/CziUC3s=', 'Content-Length': '0'} 2024-03-04T16:33:47.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074f40409ac9c0919-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.702 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.702 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.702 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.702 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.702 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.702 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.702 DEBUG:boto:Data: 2024-03-04T16:33:47.702 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.702 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.703 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.703 DEBUG:boto:Params: {} 2024-03-04T16:33:47.703 DEBUG:boto:Token: None 2024-03-04T16:33:47.703 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.703 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hZmKTnH8QvCRqRiCzYv/CziUC3s= 2024-03-04T16:33:47.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hZmKTnH8QvCRqRiCzYv/CziUC3s=', 'Content-Length': '0'} 2024-03-04T16:33:47.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000198a1c280e54f768-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.705 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.706 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.706 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.706 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.706 DEBUG:boto:Method: GET 2024-03-04T16:33:47.706 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.706 DEBUG:boto:Data: 2024-03-04T16:33:47.706 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.706 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.706 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.706 DEBUG:boto:Params: {} 2024-03-04T16:33:47.706 DEBUG:boto:Token: None 2024-03-04T16:33:47.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.707 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0= 2024-03-04T16:33:47.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0=', 'Content-Length': '0'} 2024-03-04T16:33:47.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000934d534525f8c394-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.709 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.709 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.709 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.709 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.709 DEBUG:boto:Method: GET 2024-03-04T16:33:47.709 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.709 DEBUG:boto:Data: 2024-03-04T16:33:47.710 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.710 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.710 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.710 DEBUG:boto:Params: {} 2024-03-04T16:33:47.710 DEBUG:boto:Token: None 2024-03-04T16:33:47.710 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:47.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0= 2024-03-04T16:33:47.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R1eMDIoC/C5g24KdnLmBlheMVE0=', 'Content-Length': '0'} 2024-03-04T16:33:47.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000010ffea6fdef5403-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.760 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.760 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.760 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.761 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.761 DEBUG:boto:Method: GET 2024-03-04T16:33:47.761 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.761 DEBUG:boto:Data: 2024-03-04T16:33:47.761 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.761 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.761 DEBUG:boto:Params: {} 2024-03-04T16:33:47.761 DEBUG:boto:Token: None 2024-03-04T16:33:47.761 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.762 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73f1061dd9ffd8b-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.765 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.765 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.765 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.766 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.766 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.766 DEBUG:boto:Method: GET 2024-03-04T16:33:47.766 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.766 DEBUG:boto:Data: 2024-03-04T16:33:47.766 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.766 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.766 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.766 DEBUG:boto:Params: {} 2024-03-04T16:33:47.766 DEBUG:boto:Token: None 2024-03-04T16:33:47.766 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.767 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b082048a0deab021-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.769 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.770 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.770 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.770 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.770 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.770 DEBUG:boto:Data: 2024-03-04T16:33:47.770 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.770 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.770 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.770 DEBUG:boto:Params: {} 2024-03-04T16:33:47.770 DEBUG:boto:Token: None 2024-03-04T16:33:47.771 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.771 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076527865e774c118-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.774 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.774 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.774 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.774 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:47.775 DEBUG:boto:Data: 2024-03-04T16:33:47.775 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.775 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.775 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.775 DEBUG:boto:Params: {} 2024-03-04T16:33:47.775 DEBUG:boto:Token: None 2024-03-04T16:33:47.775 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj 2024-03-04T16:33:47.775 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI= 2024-03-04T16:33:47.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:crePYCGPHeJvxJjbi8l6kAxVjXI=', 'Content-Length': '0'} 2024-03-04T16:33:47.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad768cf24e6794b7-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.779 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.779 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.779 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.779 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.779 DEBUG:boto:Method: GET 2024-03-04T16:33:47.780 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.780 DEBUG:boto:Data: 2024-03-04T16:33:47.780 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.780 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.780 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.780 DEBUG:boto:Params: {} 2024-03-04T16:33:47.780 DEBUG:boto:Token: None 2024-03-04T16:33:47.780 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.780 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060d2d7c6615e5cca-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.828 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.828 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.828 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.828 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.829 DEBUG:boto:Method: GET 2024-03-04T16:33:47.829 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.829 DEBUG:boto:Data: 2024-03-04T16:33:47.829 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.829 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.829 DEBUG:boto:Params: {} 2024-03-04T16:33:47.829 DEBUG:boto:Token: None 2024-03-04T16:33:47.829 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:47.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E= 2024-03-04T16:33:47.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:k+lsYRXNjUYo+OSLIeNV5K3hK3E=', 'Content-Length': '0'} 2024-03-04T16:33:47.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce526d55d1ca00fb-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.872 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.872 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.872 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.872 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.873 DEBUG:boto:Method: GET 2024-03-04T16:33:47.873 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.873 DEBUG:boto:Data: 2024-03-04T16:33:47.873 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.873 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.873 DEBUG:boto:Params: {} 2024-03-04T16:33:47.873 DEBUG:boto:Token: None 2024-03-04T16:33:47.873 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f2254387107a031-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.877 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.877 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.877 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.878 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.878 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.878 DEBUG:boto:Method: GET 2024-03-04T16:33:47.878 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.878 DEBUG:boto:Data: 2024-03-04T16:33:47.878 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.878 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.878 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.878 DEBUG:boto:Params: {} 2024-03-04T16:33:47.878 DEBUG:boto:Token: None 2024-03-04T16:33:47.879 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:47.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35e6fe5ac918e03-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.882 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:47.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.882 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.883 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.883 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.883 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.883 DEBUG:boto:Method: GET 2024-03-04T16:33:47.883 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.883 DEBUG:boto:Data: 2024-03-04T16:33:47.883 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.883 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.883 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.883 DEBUG:boto:Params: {} 2024-03-04T16:33:47.883 DEBUG:boto:Token: None 2024-03-04T16:33:47.884 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.884 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q= 2024-03-04T16:33:47.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q=', 'Content-Length': '0'} 2024-03-04T16:33:47.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000742f46e046f11dd0-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.932 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.932 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.933 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.933 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.933 DEBUG:boto:Method: GET 2024-03-04T16:33:47.933 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.933 DEBUG:boto:Data: 2024-03-04T16:33:47.933 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.933 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.933 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.933 DEBUG:boto:Params: {} 2024-03-04T16:33:47.933 DEBUG:boto:Token: None 2024-03-04T16:33:47.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q= 2024-03-04T16:33:47.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q=', 'Content-Length': '0'} 2024-03-04T16:33:47.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000751cb7a08c45f643-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.936 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.936 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.936 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.936 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.937 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.937 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.937 DEBUG:boto:Data: 2024-03-04T16:33:47.937 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.937 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.937 DEBUG:boto:Params: {} 2024-03-04T16:33:47.937 DEBUG:boto:Token: None 2024-03-04T16:33:47.937 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.937 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fNil2pBiMTJ2RQwlx8CdeqnXzFk= 2024-03-04T16:33:47.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fNil2pBiMTJ2RQwlx8CdeqnXzFk=', 'Content-Length': '0'} 2024-03-04T16:33:47.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046e07d6fdd15a006-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.940 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.940 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.940 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.940 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.940 DEBUG:boto:Method: HEAD 2024-03-04T16:33:47.940 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.940 DEBUG:boto:Data: 2024-03-04T16:33:47.940 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.940 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.940 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.941 DEBUG:boto:Params: {} 2024-03-04T16:33:47.941 DEBUG:boto:Token: None 2024-03-04T16:33:47.941 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.941 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fNil2pBiMTJ2RQwlx8CdeqnXzFk= 2024-03-04T16:33:47.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fNil2pBiMTJ2RQwlx8CdeqnXzFk=', 'Content-Length': '0'} 2024-03-04T16:33:47.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054adc073e3b65afc-0065e5f7eb-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:47.943 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.943 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.943 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.944 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.944 DEBUG:boto:Method: GET 2024-03-04T16:33:47.944 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.944 DEBUG:boto:Data: 2024-03-04T16:33:47.944 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.944 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.944 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.944 DEBUG:boto:Params: {} 2024-03-04T16:33:47.944 DEBUG:boto:Token: None 2024-03-04T16:33:47.944 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.944 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q= 2024-03-04T16:33:47.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q=', 'Content-Length': '0'} 2024-03-04T16:33:47.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014228c4a7fc70356-0065e5f7eb-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.992 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:47.992 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:47.992 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.993 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.993 DEBUG:boto:Method: GET 2024-03-04T16:33:47.993 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.993 DEBUG:boto:Data: 2024-03-04T16:33:47.993 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:47.993 DEBUG:boto:Port: 8000 2024-03-04T16:33:47.993 DEBUG:boto:Params: {} 2024-03-04T16:33:47.993 DEBUG:boto:Token: None 2024-03-04T16:33:47.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:47.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q= 2024-03-04T16:33:47.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IMmeO+HdkQldv2OuvKpGg5/2V7Q=', 'Content-Length': '0'} 2024-03-04T16:33:47.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ee15ce63fe0ef4-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:47.996 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:47.996 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:47.996 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.996 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:47.996 DEBUG:boto:Method: GET 2024-03-04T16:33:47.996 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:47.996 DEBUG:boto:Data: 2024-03-04T16:33:47.996 DEBUG:boto:Headers: {} 2024-03-04T16:33:47.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:47.997 DEBUG:boto:Port: 8001 2024-03-04T16:33:47.997 DEBUG:boto:Params: {} 2024-03-04T16:33:47.997 DEBUG:boto:Token: None 2024-03-04T16:33:47.997 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:47 GMT /sbnvju-22/ 2024-03-04T16:33:47.997 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI= 2024-03-04T16:33:47.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:47 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ru6yu13mn7i/sBhSSQ//dF2gnvI=', 'Content-Length': '0'} 2024-03-04T16:33:48.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003496c902dc735d1b-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.040 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.041 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.041 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.041 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.041 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.041 DEBUG:boto:Method: GET 2024-03-04T16:33:48.041 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.041 DEBUG:boto:Data: 2024-03-04T16:33:48.041 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.041 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.041 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.041 DEBUG:boto:Params: {} 2024-03-04T16:33:48.042 DEBUG:boto:Token: None 2024-03-04T16:33:48.042 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.042 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c979e2b31c407fe4-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.045 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.045 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.046 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.046 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.046 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.046 DEBUG:boto:Data: 2024-03-04T16:33:48.046 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.046 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.046 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.046 DEBUG:boto:Params: {} 2024-03-04T16:33:48.046 DEBUG:boto:Token: None 2024-03-04T16:33:48.047 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.047 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008891110735a7d8a5-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.050 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.051 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.051 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.051 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.051 DEBUG:boto:Data: 2024-03-04T16:33:48.051 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.051 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.051 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.051 DEBUG:boto:Params: {} 2024-03-04T16:33:48.051 DEBUG:boto:Token: None 2024-03-04T16:33:48.051 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.052 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008297c7d16b2206e8-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.056 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.056 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.056 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.056 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.056 DEBUG:boto:Method: GET 2024-03-04T16:33:48.056 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.056 DEBUG:boto:Data: 2024-03-04T16:33:48.057 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.057 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.058 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.058 DEBUG:boto:Params: {} 2024-03-04T16:33:48.058 DEBUG:boto:Token: None 2024-03-04T16:33:48.058 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007044b7879dc91f11-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.104 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.104 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.104 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.105 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.105 DEBUG:boto:Method: GET 2024-03-04T16:33:48.105 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.105 DEBUG:boto:Data: 2024-03-04T16:33:48.105 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.105 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.105 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.105 DEBUG:boto:Params: {} 2024-03-04T16:33:48.105 DEBUG:boto:Token: None 2024-03-04T16:33:48.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.106 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000654fc34a19ff018d-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.152 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.152 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.152 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.152 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.153 DEBUG:boto:Method: GET 2024-03-04T16:33:48.153 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.153 DEBUG:boto:Data: 2024-03-04T16:33:48.153 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.153 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.153 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.153 DEBUG:boto:Params: {} 2024-03-04T16:33:48.153 DEBUG:boto:Token: None 2024-03-04T16:33:48.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030989b0fa376761f-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.200 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.201 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.201 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.201 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.201 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.201 DEBUG:boto:Method: GET 2024-03-04T16:33:48.201 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.201 DEBUG:boto:Data: 2024-03-04T16:33:48.201 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.201 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.201 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.201 DEBUG:boto:Params: {} 2024-03-04T16:33:48.202 DEBUG:boto:Token: None 2024-03-04T16:33:48.202 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.202 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a94f0c3464b2ad-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.205 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.205 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.205 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.205 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.206 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.206 DEBUG:boto:Method: GET 2024-03-04T16:33:48.206 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.206 DEBUG:boto:Data: 2024-03-04T16:33:48.206 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.206 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.206 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.206 DEBUG:boto:Params: {} 2024-03-04T16:33:48.206 DEBUG:boto:Token: None 2024-03-04T16:33:48.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.207 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk= 2024-03-04T16:33:48.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk=', 'Content-Length': '0'} 2024-03-04T16:33:48.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6cb729e72790601-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.252 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.252 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.252 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.252 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.253 DEBUG:boto:Method: GET 2024-03-04T16:33:48.253 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.253 DEBUG:boto:Data: 2024-03-04T16:33:48.253 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.253 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.253 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.253 DEBUG:boto:Params: {} 2024-03-04T16:33:48.253 DEBUG:boto:Token: None 2024-03-04T16:33:48.253 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.253 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk= 2024-03-04T16:33:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk=', 'Content-Length': '0'} 2024-03-04T16:33:48.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041e8427ee72b3e43-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.256 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.256 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.256 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.256 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.256 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.256 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.256 DEBUG:boto:Data: 2024-03-04T16:33:48.257 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.257 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.257 DEBUG:boto:Params: {} 2024-03-04T16:33:48.257 DEBUG:boto:Token: None 2024-03-04T16:33:48.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f2DFEPe7ME4xTiQcl+NidrEamJQ= 2024-03-04T16:33:48.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f2DFEPe7ME4xTiQcl+NidrEamJQ=', 'Content-Length': '0'} 2024-03-04T16:33:48.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffe69db8fb631b19-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.259 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.260 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.260 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.260 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.260 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.260 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.260 DEBUG:boto:Data: 2024-03-04T16:33:48.260 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.260 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.260 DEBUG:boto:Params: {} 2024-03-04T16:33:48.261 DEBUG:boto:Token: None 2024-03-04T16:33:48.261 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f2DFEPe7ME4xTiQcl+NidrEamJQ= 2024-03-04T16:33:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f2DFEPe7ME4xTiQcl+NidrEamJQ=', 'Content-Length': '0'} 2024-03-04T16:33:48.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce86973e925017e5-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.263 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.263 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.263 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.263 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.263 DEBUG:boto:Method: GET 2024-03-04T16:33:48.264 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.264 DEBUG:boto:Data: 2024-03-04T16:33:48.264 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.264 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.264 DEBUG:boto:Params: {} 2024-03-04T16:33:48.264 DEBUG:boto:Token: None 2024-03-04T16:33:48.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk= 2024-03-04T16:33:48.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk=', 'Content-Length': '0'} 2024-03-04T16:33:48.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000621f673eeeea8074-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.308 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.308 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.308 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.308 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.309 DEBUG:boto:Method: GET 2024-03-04T16:33:48.309 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.309 DEBUG:boto:Data: 2024-03-04T16:33:48.309 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.309 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.309 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.309 DEBUG:boto:Params: {} 2024-03-04T16:33:48.309 DEBUG:boto:Token: None 2024-03-04T16:33:48.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:48.309 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk= 2024-03-04T16:33:48.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PKwLC/oPzpIeYMa0WgGOmpP3NXk=', 'Content-Length': '0'} 2024-03-04T16:33:48.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faf8a475c80bbfad-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.352 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.352 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.352 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.352 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.353 DEBUG:boto:Method: GET 2024-03-04T16:33:48.353 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.353 DEBUG:boto:Data: 2024-03-04T16:33:48.353 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.353 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.353 DEBUG:boto:Params: {} 2024-03-04T16:33:48.353 DEBUG:boto:Token: None 2024-03-04T16:33:48.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004e87cbb630805bc-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.404 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.405 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.405 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.405 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.405 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.405 DEBUG:boto:Method: GET 2024-03-04T16:33:48.406 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.406 DEBUG:boto:Data: 2024-03-04T16:33:48.406 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.406 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.406 DEBUG:boto:Params: {} 2024-03-04T16:33:48.406 DEBUG:boto:Token: None 2024-03-04T16:33:48.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873771d6f54a4363-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.452 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.453 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.453 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.453 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.453 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.453 DEBUG:boto:Data: 2024-03-04T16:33:48.453 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.453 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.453 DEBUG:boto:Params: {} 2024-03-04T16:33:48.454 DEBUG:boto:Token: None 2024-03-04T16:33:48.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048ba8aeb7b03fbc5-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.458 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.458 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.458 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.458 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.459 DEBUG:boto:Data: 2024-03-04T16:33:48.459 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.459 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.459 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.459 DEBUG:boto:Params: {} 2024-03-04T16:33:48.459 DEBUG:boto:Token: None 2024-03-04T16:33:48.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d9712b7bc40a5b9-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.463 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.463 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.463 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.463 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.463 DEBUG:boto:Method: GET 2024-03-04T16:33:48.463 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.463 DEBUG:boto:Data: 2024-03-04T16:33:48.463 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.464 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.464 DEBUG:boto:Params: {} 2024-03-04T16:33:48.464 DEBUG:boto:Token: None 2024-03-04T16:33:48.464 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.464 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090e9dd710fed35d9-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.508 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.508 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.509 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.509 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.509 DEBUG:boto:Method: GET 2024-03-04T16:33:48.509 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.509 DEBUG:boto:Data: 2024-03-04T16:33:48.509 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.509 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.509 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.509 DEBUG:boto:Params: {} 2024-03-04T16:33:48.509 DEBUG:boto:Token: None 2024-03-04T16:33:48.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddffc7ba8f27e142-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.556 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.556 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.556 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.557 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.557 DEBUG:boto:Method: GET 2024-03-04T16:33:48.557 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.557 DEBUG:boto:Data: 2024-03-04T16:33:48.557 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.557 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.557 DEBUG:boto:Params: {} 2024-03-04T16:33:48.557 DEBUG:boto:Token: None 2024-03-04T16:33:48.557 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca2bfacef213d9e-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.561 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.561 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.561 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.561 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.561 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.561 DEBUG:boto:Method: GET 2024-03-04T16:33:48.561 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.562 DEBUG:boto:Data: 2024-03-04T16:33:48.562 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.562 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.562 DEBUG:boto:Params: {} 2024-03-04T16:33:48.562 DEBUG:boto:Token: None 2024-03-04T16:33:48.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009960a50b4a6700b1-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.612 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.613 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.613 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.613 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.613 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.613 DEBUG:boto:Method: GET 2024-03-04T16:33:48.613 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.613 DEBUG:boto:Data: 2024-03-04T16:33:48.613 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.613 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.613 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.613 DEBUG:boto:Params: {} 2024-03-04T16:33:48.614 DEBUG:boto:Token: None 2024-03-04T16:33:48.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM= 2024-03-04T16:33:48.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM=', 'Content-Length': '0'} 2024-03-04T16:33:48.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fb1ae2ed58031cf-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.660 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.660 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.660 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.660 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.661 DEBUG:boto:Method: GET 2024-03-04T16:33:48.661 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.661 DEBUG:boto:Data: 2024-03-04T16:33:48.661 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.661 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.661 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.661 DEBUG:boto:Params: {} 2024-03-04T16:33:48.661 DEBUG:boto:Token: None 2024-03-04T16:33:48.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM= 2024-03-04T16:33:48.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM=', 'Content-Length': '0'} 2024-03-04T16:33:48.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000636730bc89c63a31-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.664 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.664 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.664 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.664 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.664 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.664 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.665 DEBUG:boto:Data: 2024-03-04T16:33:48.665 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.665 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.665 DEBUG:boto:Params: {} 2024-03-04T16:33:48.665 DEBUG:boto:Token: None 2024-03-04T16:33:48.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.665 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:df5aUTnS+sw/b8OzNU01tQk8LoE= 2024-03-04T16:33:48.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:df5aUTnS+sw/b8OzNU01tQk8LoE=', 'Content-Length': '0'} 2024-03-04T16:33:48.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dc1e8b63423bcf6-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.667 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.667 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.667 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.668 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.668 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.668 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.668 DEBUG:boto:Data: 2024-03-04T16:33:48.668 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.668 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.668 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.668 DEBUG:boto:Params: {} 2024-03-04T16:33:48.668 DEBUG:boto:Token: None 2024-03-04T16:33:48.668 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:df5aUTnS+sw/b8OzNU01tQk8LoE= 2024-03-04T16:33:48.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:df5aUTnS+sw/b8OzNU01tQk8LoE=', 'Content-Length': '0'} 2024-03-04T16:33:48.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2e2b17db65e93b2-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.671 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.671 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.671 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.671 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.671 DEBUG:boto:Method: GET 2024-03-04T16:33:48.671 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.671 DEBUG:boto:Data: 2024-03-04T16:33:48.671 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.672 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.672 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.672 DEBUG:boto:Params: {} 2024-03-04T16:33:48.672 DEBUG:boto:Token: None 2024-03-04T16:33:48.672 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.672 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM= 2024-03-04T16:33:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM=', 'Content-Length': '0'} 2024-03-04T16:33:48.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b65b81ef8a07a4e-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.674 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.674 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.675 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.675 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.675 DEBUG:boto:Method: GET 2024-03-04T16:33:48.675 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.675 DEBUG:boto:Data: 2024-03-04T16:33:48.675 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.675 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.675 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.675 DEBUG:boto:Params: {} 2024-03-04T16:33:48.675 DEBUG:boto:Token: None 2024-03-04T16:33:48.675 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:48.676 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM= 2024-03-04T16:33:48.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SlVyh5q0q/H+btkUx9Q+AY8t7AM=', 'Content-Length': '0'} 2024-03-04T16:33:48.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000440fe82f506a76fe-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.720 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.721 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.721 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.721 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.721 DEBUG:boto:Method: GET 2024-03-04T16:33:48.721 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.721 DEBUG:boto:Data: 2024-03-04T16:33:48.722 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.722 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.722 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.722 DEBUG:boto:Params: {} 2024-03-04T16:33:48.722 DEBUG:boto:Token: None 2024-03-04T16:33:48.723 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.723 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff5eccb1d42a1988-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.727 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.728 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.728 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.728 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.728 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.729 DEBUG:boto:Method: GET 2024-03-04T16:33:48.729 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.729 DEBUG:boto:Data: 2024-03-04T16:33:48.729 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.729 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.729 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.729 DEBUG:boto:Params: {} 2024-03-04T16:33:48.730 DEBUG:boto:Token: None 2024-03-04T16:33:48.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb42b489e3d5acb-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.733 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.734 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.734 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.734 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.734 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.734 DEBUG:boto:Data: 2024-03-04T16:33:48.734 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.735 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.735 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.735 DEBUG:boto:Params: {} 2024-03-04T16:33:48.735 DEBUG:boto:Token: None 2024-03-04T16:33:48.735 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.735 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000748d41a8f97035a5-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.739 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.739 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.739 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.739 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:48.739 DEBUG:boto:Data: 2024-03-04T16:33:48.739 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.740 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.740 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.740 DEBUG:boto:Params: {} 2024-03-04T16:33:48.740 DEBUG:boto:Token: None 2024-03-04T16:33:48.740 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj 2024-03-04T16:33:48.740 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE= 2024-03-04T16:33:48.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+2FKKiScbwkuN/KlY9C0IvPtbcE=', 'Content-Length': '0'} 2024-03-04T16:33:48.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b8272367427a797-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.744 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.744 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.744 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.744 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.744 DEBUG:boto:Method: GET 2024-03-04T16:33:48.745 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.745 DEBUG:boto:Data: 2024-03-04T16:33:48.745 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.745 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.745 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.745 DEBUG:boto:Params: {} 2024-03-04T16:33:48.745 DEBUG:boto:Token: None 2024-03-04T16:33:48.745 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.745 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d825715da3e2056-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.788 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.788 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.788 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.789 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.789 DEBUG:boto:Method: GET 2024-03-04T16:33:48.789 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.789 DEBUG:boto:Data: 2024-03-04T16:33:48.789 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.789 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.789 DEBUG:boto:Params: {} 2024-03-04T16:33:48.789 DEBUG:boto:Token: None 2024-03-04T16:33:48.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:48.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8= 2024-03-04T16:33:48.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yF8DQgxKf/8StLDNHy2ZBR1mbB8=', 'Content-Length': '0'} 2024-03-04T16:33:48.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b16353011a32446e-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.840 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.840 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.840 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.840 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.841 DEBUG:boto:Method: GET 2024-03-04T16:33:48.841 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.841 DEBUG:boto:Data: 2024-03-04T16:33:48.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.841 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.841 DEBUG:boto:Params: {} 2024-03-04T16:33:48.841 DEBUG:boto:Token: None 2024-03-04T16:33:48.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099895ea72d277bf3-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.888 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.889 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:48.889 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:48.889 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.889 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:48.889 DEBUG:boto:Method: GET 2024-03-04T16:33:48.889 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:48.889 DEBUG:boto:Data: 2024-03-04T16:33:48.889 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.889 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.889 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.889 DEBUG:boto:Params: {} 2024-03-04T16:33:48.890 DEBUG:boto:Token: None 2024-03-04T16:33:48.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/ 2024-03-04T16:33:48.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I= 2024-03-04T16:33:48.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9Gtmmbd7rrA2f5YSyFB8kv21Y5I=', 'Content-Length': '0'} 2024-03-04T16:33:48.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b3b6c052786090-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.936 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:48.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.937 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.937 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.937 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.937 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.937 DEBUG:boto:Method: GET 2024-03-04T16:33:48.937 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.937 DEBUG:boto:Data: 2024-03-04T16:33:48.937 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.937 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.938 DEBUG:boto:Params: {} 2024-03-04T16:33:48.938 DEBUG:boto:Token: None 2024-03-04T16:33:48.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E= 2024-03-04T16:33:48.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E=', 'Content-Length': '0'} 2024-03-04T16:33:48.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cfb1949de868cc4-0065e5f7ec-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.984 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.984 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.984 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.985 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.985 DEBUG:boto:Method: GET 2024-03-04T16:33:48.985 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.985 DEBUG:boto:Data: 2024-03-04T16:33:48.985 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.985 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.985 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.985 DEBUG:boto:Params: {} 2024-03-04T16:33:48.985 DEBUG:boto:Token: None 2024-03-04T16:33:48.986 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.986 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E= 2024-03-04T16:33:48.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E=', 'Content-Length': '0'} 2024-03-04T16:33:48.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057b53c391835f409-0065e5f7ec-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.988 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.988 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.988 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.989 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.989 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.989 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.989 DEBUG:boto:Data: 2024-03-04T16:33:48.989 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.989 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.989 DEBUG:boto:Params: {} 2024-03-04T16:33:48.990 DEBUG:boto:Token: None 2024-03-04T16:33:48.990 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IzFTB1jJ8S5bph+Y9RmV8rKqQ/s= 2024-03-04T16:33:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IzFTB1jJ8S5bph+Y9RmV8rKqQ/s=', 'Content-Length': '0'} 2024-03-04T16:33:48.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001642b5a1123a3c05-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.993 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.993 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.993 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.993 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.993 DEBUG:boto:Method: HEAD 2024-03-04T16:33:48.993 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.993 DEBUG:boto:Data: 2024-03-04T16:33:48.994 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.994 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:48.994 DEBUG:boto:Port: 8000 2024-03-04T16:33:48.994 DEBUG:boto:Params: {} 2024-03-04T16:33:48.994 DEBUG:boto:Token: None 2024-03-04T16:33:48.994 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.994 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IzFTB1jJ8S5bph+Y9RmV8rKqQ/s= 2024-03-04T16:33:48.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IzFTB1jJ8S5bph+Y9RmV8rKqQ/s=', 'Content-Length': '0'} 2024-03-04T16:33:48.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d453258eec44729d-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:48.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:48.997 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:48.997 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:48.997 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.997 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.997 DEBUG:boto:Method: GET 2024-03-04T16:33:48.997 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.998 DEBUG:boto:Data: 2024-03-04T16:33:48.998 DEBUG:boto:Headers: {} 2024-03-04T16:33:48.998 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:48.998 DEBUG:boto:Port: 8001 2024-03-04T16:33:48.998 DEBUG:boto:Params: {} 2024-03-04T16:33:48.998 DEBUG:boto:Token: None 2024-03-04T16:33:48.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:48 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:48.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E= 2024-03-04T16:33:48.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z7vr3syzISxp6S61MzWgYTY0K0E=', 'Content-Length': '0'} 2024-03-04T16:33:49.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086b09d0dcf6da419-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.044 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.044 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.044 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:49.045 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:49.045 DEBUG:boto:Method: GET 2024-03-04T16:33:49.045 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:49.045 DEBUG:boto:Data: 2024-03-04T16:33:49.045 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.045 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.045 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.045 DEBUG:boto:Params: {} 2024-03-04T16:33:49.045 DEBUG:boto:Token: None 2024-03-04T16:33:49.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:49.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PEdF2o2OtNC0pNjbNECGeF1KsWU= 2024-03-04T16:33:49.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PEdF2o2OtNC0pNjbNECGeF1KsWU=', 'Content-Length': '0'} 2024-03-04T16:33:49.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bafaa49e4ec7c08-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.092 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.092 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.093 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.093 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.093 DEBUG:boto:Method: GET 2024-03-04T16:33:49.093 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.093 DEBUG:boto:Data: 2024-03-04T16:33:49.094 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.094 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.094 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.094 DEBUG:boto:Params: {} 2024-03-04T16:33:49.094 DEBUG:boto:Token: None 2024-03-04T16:33:49.095 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.095 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd5f9972492b1419-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.140 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.141 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.141 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.141 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.142 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.142 DEBUG:boto:Method: GET 2024-03-04T16:33:49.142 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.142 DEBUG:boto:Data: 2024-03-04T16:33:49.142 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.142 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.142 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.142 DEBUG:boto:Params: {} 2024-03-04T16:33:49.142 DEBUG:boto:Token: None 2024-03-04T16:33:49.142 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.143 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651d63f044a46355-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.146 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.147 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.148 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.148 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.148 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.148 DEBUG:boto:Data: 2024-03-04T16:33:49.148 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.148 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.148 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.148 DEBUG:boto:Params: {} 2024-03-04T16:33:49.149 DEBUG:boto:Token: None 2024-03-04T16:33:49.149 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000727dcf9cf99d1810-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.153 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.153 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.153 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.154 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.154 DEBUG:boto:Data: 2024-03-04T16:33:49.154 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.154 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.154 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.154 DEBUG:boto:Params: {} 2024-03-04T16:33:49.154 DEBUG:boto:Token: None 2024-03-04T16:33:49.154 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.155 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075ad4f31b3a13b67-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.158 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.159 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.159 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.159 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.159 DEBUG:boto:Method: GET 2024-03-04T16:33:49.159 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.159 DEBUG:boto:Data: 2024-03-04T16:33:49.159 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.159 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.159 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.159 DEBUG:boto:Params: {} 2024-03-04T16:33:49.159 DEBUG:boto:Token: None 2024-03-04T16:33:49.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.160 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c905bfe20cdd7c03-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.209 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.209 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.209 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.209 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.209 DEBUG:boto:Method: GET 2024-03-04T16:33:49.209 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.209 DEBUG:boto:Data: 2024-03-04T16:33:49.209 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.210 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.210 DEBUG:boto:Params: {} 2024-03-04T16:33:49.210 DEBUG:boto:Token: None 2024-03-04T16:33:49.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a437fbc99e099a61-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.256 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.256 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.256 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.256 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.257 DEBUG:boto:Method: GET 2024-03-04T16:33:49.257 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.257 DEBUG:boto:Data: 2024-03-04T16:33:49.257 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.257 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.257 DEBUG:boto:Params: {} 2024-03-04T16:33:49.257 DEBUG:boto:Token: None 2024-03-04T16:33:49.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d741c872e2b5e98-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.304 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.305 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.305 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.305 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.305 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.305 DEBUG:boto:Method: GET 2024-03-04T16:33:49.305 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.305 DEBUG:boto:Data: 2024-03-04T16:33:49.305 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.305 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.305 DEBUG:boto:Params: {} 2024-03-04T16:33:49.306 DEBUG:boto:Token: None 2024-03-04T16:33:49.306 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2f2293ba5487f9-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.309 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.310 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.310 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.310 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.310 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.310 DEBUG:boto:Method: GET 2024-03-04T16:33:49.310 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.310 DEBUG:boto:Data: 2024-03-04T16:33:49.310 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.310 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.311 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.311 DEBUG:boto:Params: {} 2024-03-04T16:33:49.311 DEBUG:boto:Token: None 2024-03-04T16:33:49.311 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0= 2024-03-04T16:33:49.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0=', 'Content-Length': '0'} 2024-03-04T16:33:49.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054b866f3d9fdfc07-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.356 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.357 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.357 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.357 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.357 DEBUG:boto:Method: GET 2024-03-04T16:33:49.357 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.358 DEBUG:boto:Data: 2024-03-04T16:33:49.358 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.358 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.358 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.358 DEBUG:boto:Params: {} 2024-03-04T16:33:49.359 DEBUG:boto:Token: None 2024-03-04T16:33:49.359 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.359 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0= 2024-03-04T16:33:49.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0=', 'Content-Length': '0'} 2024-03-04T16:33:49.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057ac15e6e6f4ea4c-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.362 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.363 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.363 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.363 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.363 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.363 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.364 DEBUG:boto:Data: 2024-03-04T16:33:49.364 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.364 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.364 DEBUG:boto:Params: {} 2024-03-04T16:33:49.365 DEBUG:boto:Token: None 2024-03-04T16:33:49.365 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Kx8QJEZgmWdqYOd6HMYuI45OGAA= 2024-03-04T16:33:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Kx8QJEZgmWdqYOd6HMYuI45OGAA=', 'Content-Length': '0'} 2024-03-04T16:33:49.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5c15eb7fd6050cf-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.369 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.369 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.369 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.370 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.370 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.370 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.370 DEBUG:boto:Data: 2024-03-04T16:33:49.370 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.370 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.371 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.371 DEBUG:boto:Params: {} 2024-03-04T16:33:49.371 DEBUG:boto:Token: None 2024-03-04T16:33:49.371 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.372 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Kx8QJEZgmWdqYOd6HMYuI45OGAA= 2024-03-04T16:33:49.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Kx8QJEZgmWdqYOd6HMYuI45OGAA=', 'Content-Length': '0'} 2024-03-04T16:33:49.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003471d7e5ea63d393-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.375 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.375 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.375 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.376 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.376 DEBUG:boto:Method: GET 2024-03-04T16:33:49.376 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.377 DEBUG:boto:Data: 2024-03-04T16:33:49.377 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.377 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.377 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.377 DEBUG:boto:Params: {} 2024-03-04T16:33:49.378 DEBUG:boto:Token: None 2024-03-04T16:33:49.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0= 2024-03-04T16:33:49.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0=', 'Content-Length': '0'} 2024-03-04T16:33:49.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c3e3b1d423df01e-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.428 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.428 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.429 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.429 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.429 DEBUG:boto:Method: GET 2024-03-04T16:33:49.429 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.429 DEBUG:boto:Data: 2024-03-04T16:33:49.429 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.430 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.430 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.430 DEBUG:boto:Params: {} 2024-03-04T16:33:49.430 DEBUG:boto:Token: None 2024-03-04T16:33:49.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:49.431 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0= 2024-03-04T16:33:49.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zgNGj404fFd5MrQ8Oc379O9H+/0=', 'Content-Length': '0'} 2024-03-04T16:33:49.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f982f217a1c01502-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.435 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.435 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.435 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.435 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.435 DEBUG:boto:Method: GET 2024-03-04T16:33:49.435 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.435 DEBUG:boto:Data: 2024-03-04T16:33:49.435 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.436 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.436 DEBUG:boto:Params: {} 2024-03-04T16:33:49.436 DEBUG:boto:Token: None 2024-03-04T16:33:49.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.436 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016921d7e4b549ade-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.439 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.440 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.440 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.440 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.440 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.440 DEBUG:boto:Method: GET 2024-03-04T16:33:49.440 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.440 DEBUG:boto:Data: 2024-03-04T16:33:49.441 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.441 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.441 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.441 DEBUG:boto:Params: {} 2024-03-04T16:33:49.441 DEBUG:boto:Token: None 2024-03-04T16:33:49.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f95f30b7cd0348-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.444 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.445 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.445 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.445 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.445 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.445 DEBUG:boto:Data: 2024-03-04T16:33:49.445 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.445 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.445 DEBUG:boto:Params: {} 2024-03-04T16:33:49.446 DEBUG:boto:Token: None 2024-03-04T16:33:49.446 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4cacf284aa505f7-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.450 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.450 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.450 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.450 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.450 DEBUG:boto:Data: 2024-03-04T16:33:49.450 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.450 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.450 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.450 DEBUG:boto:Params: {} 2024-03-04T16:33:49.451 DEBUG:boto:Token: None 2024-03-04T16:33:49.451 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.451 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e45dc18ffd4d81d-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.455 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.455 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.455 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.455 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.455 DEBUG:boto:Method: GET 2024-03-04T16:33:49.455 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.455 DEBUG:boto:Data: 2024-03-04T16:33:49.455 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.455 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.455 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.456 DEBUG:boto:Params: {} 2024-03-04T16:33:49.456 DEBUG:boto:Token: None 2024-03-04T16:33:49.456 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.456 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f55b9822cb73214-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.500 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.500 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.500 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.501 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.501 DEBUG:boto:Method: GET 2024-03-04T16:33:49.501 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.501 DEBUG:boto:Data: 2024-03-04T16:33:49.501 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.501 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.501 DEBUG:boto:Params: {} 2024-03-04T16:33:49.501 DEBUG:boto:Token: None 2024-03-04T16:33:49.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ed3edc53befe165-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.548 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.548 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.548 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.549 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.549 DEBUG:boto:Method: GET 2024-03-04T16:33:49.549 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.549 DEBUG:boto:Data: 2024-03-04T16:33:49.549 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.549 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.549 DEBUG:boto:Params: {} 2024-03-04T16:33:49.549 DEBUG:boto:Token: None 2024-03-04T16:33:49.550 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801e82d8a5c8eb0a-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.596 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.597 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.597 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.597 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.597 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.597 DEBUG:boto:Method: GET 2024-03-04T16:33:49.597 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.597 DEBUG:boto:Data: 2024-03-04T16:33:49.597 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.598 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.598 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.598 DEBUG:boto:Params: {} 2024-03-04T16:33:49.598 DEBUG:boto:Token: None 2024-03-04T16:33:49.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1088e7f0d66f07d-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.602 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.602 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.602 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.602 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.602 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.603 DEBUG:boto:Method: GET 2024-03-04T16:33:49.603 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.603 DEBUG:boto:Data: 2024-03-04T16:33:49.603 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.603 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.603 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.603 DEBUG:boto:Params: {} 2024-03-04T16:33:49.603 DEBUG:boto:Token: None 2024-03-04T16:33:49.603 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.603 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU= 2024-03-04T16:33:49.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU=', 'Content-Length': '0'} 2024-03-04T16:33:49.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbd486880c4a4fc2-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.649 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.649 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.649 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.649 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.649 DEBUG:boto:Method: GET 2024-03-04T16:33:49.649 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.649 DEBUG:boto:Data: 2024-03-04T16:33:49.649 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.649 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.650 DEBUG:boto:Params: {} 2024-03-04T16:33:49.650 DEBUG:boto:Token: None 2024-03-04T16:33:49.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU= 2024-03-04T16:33:49.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU=', 'Content-Length': '0'} 2024-03-04T16:33:49.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015ce2a414d381373-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.653 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.653 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.653 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.654 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.654 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.654 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.654 DEBUG:boto:Data: 2024-03-04T16:33:49.654 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.654 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.654 DEBUG:boto:Params: {} 2024-03-04T16:33:49.654 DEBUG:boto:Token: None 2024-03-04T16:33:49.655 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.655 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tK2YO5d67AET0zFPlXkrZL32sks= 2024-03-04T16:33:49.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tK2YO5d67AET0zFPlXkrZL32sks=', 'Content-Length': '0'} 2024-03-04T16:33:49.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fc3d498c466110c-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.657 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.657 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.657 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.657 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.657 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.657 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.658 DEBUG:boto:Data: 2024-03-04T16:33:49.658 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.658 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.658 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.658 DEBUG:boto:Params: {} 2024-03-04T16:33:49.658 DEBUG:boto:Token: None 2024-03-04T16:33:49.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.658 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tK2YO5d67AET0zFPlXkrZL32sks= 2024-03-04T16:33:49.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tK2YO5d67AET0zFPlXkrZL32sks=', 'Content-Length': '0'} 2024-03-04T16:33:49.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006690a59f2a91ee68-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.661 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.661 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.661 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.661 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.661 DEBUG:boto:Method: GET 2024-03-04T16:33:49.661 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.661 DEBUG:boto:Data: 2024-03-04T16:33:49.661 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.661 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.661 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.662 DEBUG:boto:Params: {} 2024-03-04T16:33:49.662 DEBUG:boto:Token: None 2024-03-04T16:33:49.662 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU= 2024-03-04T16:33:49.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU=', 'Content-Length': '0'} 2024-03-04T16:33:49.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd1872d04bd544a2-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.712 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.712 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.712 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.712 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.713 DEBUG:boto:Method: GET 2024-03-04T16:33:49.713 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.713 DEBUG:boto:Data: 2024-03-04T16:33:49.713 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.713 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.713 DEBUG:boto:Params: {} 2024-03-04T16:33:49.713 DEBUG:boto:Token: None 2024-03-04T16:33:49.713 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:49.713 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU= 2024-03-04T16:33:49.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Hcjql/HqgKfu8ZKiMIxRNktEDVU=', 'Content-Length': '0'} 2024-03-04T16:33:49.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffdb5621327de2c3-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.716 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.716 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.716 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.716 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.716 DEBUG:boto:Method: GET 2024-03-04T16:33:49.717 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.717 DEBUG:boto:Data: 2024-03-04T16:33:49.717 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.717 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.717 DEBUG:boto:Params: {} 2024-03-04T16:33:49.717 DEBUG:boto:Token: None 2024-03-04T16:33:49.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.717 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b659c11623c2da97-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.720 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.721 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.721 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.721 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.721 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.721 DEBUG:boto:Method: GET 2024-03-04T16:33:49.722 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.722 DEBUG:boto:Data: 2024-03-04T16:33:49.722 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.722 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.722 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.722 DEBUG:boto:Params: {} 2024-03-04T16:33:49.722 DEBUG:boto:Token: None 2024-03-04T16:33:49.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629fa87b355f51f5-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.725 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.726 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.726 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.726 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.726 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.726 DEBUG:boto:Data: 2024-03-04T16:33:49.726 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.726 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.726 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.726 DEBUG:boto:Params: {} 2024-03-04T16:33:49.726 DEBUG:boto:Token: None 2024-03-04T16:33:49.727 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f4da6368881fbb9-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.730 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.730 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.730 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.730 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:49.731 DEBUG:boto:Data: 2024-03-04T16:33:49.731 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.731 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.731 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.731 DEBUG:boto:Params: {} 2024-03-04T16:33:49.731 DEBUG:boto:Token: None 2024-03-04T16:33:49.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj 2024-03-04T16:33:49.732 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE= 2024-03-04T16:33:49.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xEcpCZhgziZfbDO7962vFHMK0GE=', 'Content-Length': '0'} 2024-03-04T16:33:49.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3060bbbb7c30b1b-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.735 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.735 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.735 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.735 DEBUG:boto:Method: GET 2024-03-04T16:33:49.736 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.736 DEBUG:boto:Data: 2024-03-04T16:33:49.736 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.736 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.736 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.736 DEBUG:boto:Params: {} 2024-03-04T16:33:49.736 DEBUG:boto:Token: None 2024-03-04T16:33:49.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3eb000762f4f078-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.780 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.781 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.781 DEBUG:boto:Method: GET 2024-03-04T16:33:49.781 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.781 DEBUG:boto:Data: 2024-03-04T16:33:49.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.781 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.781 DEBUG:boto:Params: {} 2024-03-04T16:33:49.781 DEBUG:boto:Token: None 2024-03-04T16:33:49.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:49.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM= 2024-03-04T16:33:49.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5heqFQxTAJGWpOlYFMF+FCoAADM=', 'Content-Length': '0'} 2024-03-04T16:33:49.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028d1d82bae476c3d-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.828 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.828 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.828 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.829 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.829 DEBUG:boto:Method: GET 2024-03-04T16:33:49.829 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.829 DEBUG:boto:Data: 2024-03-04T16:33:49.829 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.829 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.829 DEBUG:boto:Params: {} 2024-03-04T16:33:49.829 DEBUG:boto:Token: None 2024-03-04T16:33:49.830 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd708dc7fb21687-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.833 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.833 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:49.833 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:49.834 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.834 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:49.834 DEBUG:boto:Method: GET 2024-03-04T16:33:49.834 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:49.834 DEBUG:boto:Data: 2024-03-04T16:33:49.834 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.834 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.834 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.834 DEBUG:boto:Params: {} 2024-03-04T16:33:49.835 DEBUG:boto:Token: None 2024-03-04T16:33:49.835 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/ 2024-03-04T16:33:49.835 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I= 2024-03-04T16:33:49.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brElCU94KFiWL0VjPNgUmhhkX4I=', 'Content-Length': '0'} 2024-03-04T16:33:49.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000621e6ff9f01cc4b-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.884 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:49.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.885 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.885 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.885 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.885 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.885 DEBUG:boto:Method: GET 2024-03-04T16:33:49.885 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.885 DEBUG:boto:Data: 2024-03-04T16:33:49.886 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.886 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.886 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.886 DEBUG:boto:Params: {} 2024-03-04T16:33:49.886 DEBUG:boto:Token: None 2024-03-04T16:33:49.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o= 2024-03-04T16:33:49.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o=', 'Content-Length': '0'} 2024-03-04T16:33:49.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b57c133d73b8cf9f-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.937 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.937 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.937 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.937 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.937 DEBUG:boto:Method: GET 2024-03-04T16:33:49.937 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.937 DEBUG:boto:Data: 2024-03-04T16:33:49.937 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.937 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.937 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.938 DEBUG:boto:Params: {} 2024-03-04T16:33:49.938 DEBUG:boto:Token: None 2024-03-04T16:33:49.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o= 2024-03-04T16:33:49.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o=', 'Content-Length': '0'} 2024-03-04T16:33:49.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000d9438aec5da80d-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.941 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.941 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.941 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.941 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.941 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.941 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.941 DEBUG:boto:Data: 2024-03-04T16:33:49.941 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.942 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.942 DEBUG:boto:Params: {} 2024-03-04T16:33:49.942 DEBUG:boto:Token: None 2024-03-04T16:33:49.942 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ud7SzgrAC3luXKrA4yNU7YdJZuE= 2024-03-04T16:33:49.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ud7SzgrAC3luXKrA4yNU7YdJZuE=', 'Content-Length': '0'} 2024-03-04T16:33:49.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ea2dc4871ce9098-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.945 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.945 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.945 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.945 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.945 DEBUG:boto:Method: HEAD 2024-03-04T16:33:49.945 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.945 DEBUG:boto:Data: 2024-03-04T16:33:49.946 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.946 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:49.946 DEBUG:boto:Port: 8000 2024-03-04T16:33:49.946 DEBUG:boto:Params: {} 2024-03-04T16:33:49.946 DEBUG:boto:Token: None 2024-03-04T16:33:49.946 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ud7SzgrAC3luXKrA4yNU7YdJZuE= 2024-03-04T16:33:49.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ud7SzgrAC3luXKrA4yNU7YdJZuE=', 'Content-Length': '0'} 2024-03-04T16:33:49.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b59ccdb83c9ba84f-0065e5f7ed-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:49.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:49.949 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:49.949 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:49.949 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.949 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.949 DEBUG:boto:Method: GET 2024-03-04T16:33:49.950 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.950 DEBUG:boto:Data: 2024-03-04T16:33:49.950 DEBUG:boto:Headers: {} 2024-03-04T16:33:49.950 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:49.950 DEBUG:boto:Port: 8001 2024-03-04T16:33:49.950 DEBUG:boto:Params: {} 2024-03-04T16:33:49.950 DEBUG:boto:Token: None 2024-03-04T16:33:49.950 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:49 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:49.950 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o= 2024-03-04T16:33:49.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Qr6TP9xXx9FqK0o5xSMXVec0X7o=', 'Content-Length': '0'} 2024-03-04T16:33:49.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008355c7ed5c1c63c3-0065e5f7ed-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.000 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.000 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.000 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:50.001 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:50.001 DEBUG:boto:Method: GET 2024-03-04T16:33:50.001 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:50.001 DEBUG:boto:Data: 2024-03-04T16:33:50.001 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.001 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.001 DEBUG:boto:Params: {} 2024-03-04T16:33:50.001 DEBUG:boto:Token: None 2024-03-04T16:33:50.001 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:50.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cjqg1+Mynv2myj3klmw6irNnX/g= 2024-03-04T16:33:50.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cjqg1+Mynv2myj3klmw6irNnX/g=', 'Content-Length': '0'} 2024-03-04T16:33:50.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022f4efe87191b836-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.048 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.048 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.048 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.049 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.049 DEBUG:boto:Method: GET 2024-03-04T16:33:50.049 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.049 DEBUG:boto:Data: 2024-03-04T16:33:50.049 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.049 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.049 DEBUG:boto:Params: {} 2024-03-04T16:33:50.050 DEBUG:boto:Token: None 2024-03-04T16:33:50.050 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278122b500175323-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.100 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.101 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.101 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.101 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.101 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.101 DEBUG:boto:Method: GET 2024-03-04T16:33:50.101 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.101 DEBUG:boto:Data: 2024-03-04T16:33:50.101 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.101 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.102 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.102 DEBUG:boto:Params: {} 2024-03-04T16:33:50.102 DEBUG:boto:Token: None 2024-03-04T16:33:50.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000013f07139cd74910-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.106 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.106 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.106 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.106 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.106 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.106 DEBUG:boto:Data: 2024-03-04T16:33:50.107 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.107 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.107 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.107 DEBUG:boto:Params: {} 2024-03-04T16:33:50.107 DEBUG:boto:Token: None 2024-03-04T16:33:50.107 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.107 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006aff81f42d5d2286-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.111 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.111 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.111 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.112 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.112 DEBUG:boto:Data: 2024-03-04T16:33:50.112 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.112 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.112 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.112 DEBUG:boto:Params: {} 2024-03-04T16:33:50.112 DEBUG:boto:Token: None 2024-03-04T16:33:50.112 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.112 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c9687fbab194fc2-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.116 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.116 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.116 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.116 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.116 DEBUG:boto:Method: GET 2024-03-04T16:33:50.116 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.116 DEBUG:boto:Data: 2024-03-04T16:33:50.117 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.117 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.117 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.117 DEBUG:boto:Params: {} 2024-03-04T16:33:50.117 DEBUG:boto:Token: None 2024-03-04T16:33:50.117 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009791709a3cac032c-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.160 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.160 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.161 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.161 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.161 DEBUG:boto:Method: GET 2024-03-04T16:33:50.161 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.161 DEBUG:boto:Data: 2024-03-04T16:33:50.161 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.161 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.161 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.161 DEBUG:boto:Params: {} 2024-03-04T16:33:50.161 DEBUG:boto:Token: None 2024-03-04T16:33:50.162 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.162 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb1bd1c4b265f398-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.212 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.212 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.212 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.212 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.213 DEBUG:boto:Method: GET 2024-03-04T16:33:50.213 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.213 DEBUG:boto:Data: 2024-03-04T16:33:50.213 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.213 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.213 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.213 DEBUG:boto:Params: {} 2024-03-04T16:33:50.213 DEBUG:boto:Token: None 2024-03-04T16:33:50.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.213 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5654c38854c5d35-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.260 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.261 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.261 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.261 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.261 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.261 DEBUG:boto:Method: GET 2024-03-04T16:33:50.261 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.261 DEBUG:boto:Data: 2024-03-04T16:33:50.261 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.261 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.261 DEBUG:boto:Params: {} 2024-03-04T16:33:50.262 DEBUG:boto:Token: None 2024-03-04T16:33:50.262 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.262 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6cd91874911a12-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.265 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.266 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.266 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.266 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.266 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.266 DEBUG:boto:Method: GET 2024-03-04T16:33:50.266 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.266 DEBUG:boto:Data: 2024-03-04T16:33:50.266 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.267 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.267 DEBUG:boto:Params: {} 2024-03-04T16:33:50.267 DEBUG:boto:Token: None 2024-03-04T16:33:50.267 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU= 2024-03-04T16:33:50.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU=', 'Content-Length': '0'} 2024-03-04T16:33:50.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8b540f484a5a502-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.269 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.269 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.270 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.270 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.270 DEBUG:boto:Method: GET 2024-03-04T16:33:50.270 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.270 DEBUG:boto:Data: 2024-03-04T16:33:50.270 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.270 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.270 DEBUG:boto:Params: {} 2024-03-04T16:33:50.270 DEBUG:boto:Token: None 2024-03-04T16:33:50.271 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU= 2024-03-04T16:33:50.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU=', 'Content-Length': '0'} 2024-03-04T16:33:50.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4d8143758fb96d-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.273 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.274 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.274 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.274 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.274 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.274 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.274 DEBUG:boto:Data: 2024-03-04T16:33:50.274 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.275 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.275 DEBUG:boto:Params: {} 2024-03-04T16:33:50.275 DEBUG:boto:Token: None 2024-03-04T16:33:50.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.276 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sbtWU3wM+nCDdQeGofKMaGEbjGI= 2024-03-04T16:33:50.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sbtWU3wM+nCDdQeGofKMaGEbjGI=', 'Content-Length': '0'} 2024-03-04T16:33:50.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066289eadd0f257ae-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.278 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.278 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.278 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.278 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.278 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.278 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.278 DEBUG:boto:Data: 2024-03-04T16:33:50.278 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.278 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.278 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.278 DEBUG:boto:Params: {} 2024-03-04T16:33:50.279 DEBUG:boto:Token: None 2024-03-04T16:33:50.279 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.279 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sbtWU3wM+nCDdQeGofKMaGEbjGI= 2024-03-04T16:33:50.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sbtWU3wM+nCDdQeGofKMaGEbjGI=', 'Content-Length': '0'} 2024-03-04T16:33:50.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b6d4b3e5b191959-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.281 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.281 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.281 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.281 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.281 DEBUG:boto:Method: GET 2024-03-04T16:33:50.282 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.282 DEBUG:boto:Data: 2024-03-04T16:33:50.282 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.282 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.282 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.282 DEBUG:boto:Params: {} 2024-03-04T16:33:50.282 DEBUG:boto:Token: None 2024-03-04T16:33:50.282 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.282 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU= 2024-03-04T16:33:50.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU=', 'Content-Length': '0'} 2024-03-04T16:33:50.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045437f3b0f9f45b2-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.284 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.284 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.285 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.285 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.285 DEBUG:boto:Method: GET 2024-03-04T16:33:50.285 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.285 DEBUG:boto:Data: 2024-03-04T16:33:50.285 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.285 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.285 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.285 DEBUG:boto:Params: {} 2024-03-04T16:33:50.285 DEBUG:boto:Token: None 2024-03-04T16:33:50.285 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:50.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU= 2024-03-04T16:33:50.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DRxmbFxnAaMlXhjrceqRQjF0qsU=', 'Content-Length': '0'} 2024-03-04T16:33:50.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f559ca81deb4d10-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.328 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.328 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.328 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.329 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.329 DEBUG:boto:Method: GET 2024-03-04T16:33:50.329 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.329 DEBUG:boto:Data: 2024-03-04T16:33:50.329 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.329 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.329 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.329 DEBUG:boto:Params: {} 2024-03-04T16:33:50.329 DEBUG:boto:Token: None 2024-03-04T16:33:50.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2409236517f8f68-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.333 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.333 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.333 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.333 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.334 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.334 DEBUG:boto:Method: GET 2024-03-04T16:33:50.334 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.334 DEBUG:boto:Data: 2024-03-04T16:33:50.334 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.334 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.334 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.334 DEBUG:boto:Params: {} 2024-03-04T16:33:50.334 DEBUG:boto:Token: None 2024-03-04T16:33:50.334 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.335 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032afd10c1ecc48c1-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.338 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.338 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.338 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.338 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.338 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.338 DEBUG:boto:Data: 2024-03-04T16:33:50.338 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.338 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.339 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.339 DEBUG:boto:Params: {} 2024-03-04T16:33:50.339 DEBUG:boto:Token: None 2024-03-04T16:33:50.339 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.339 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005235f60ae0821b1a-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.343 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.343 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.343 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.343 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.343 DEBUG:boto:Data: 2024-03-04T16:33:50.343 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.343 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.344 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.344 DEBUG:boto:Params: {} 2024-03-04T16:33:50.344 DEBUG:boto:Token: None 2024-03-04T16:33:50.344 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.344 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daddae32b43be4bb-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.348 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.348 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.348 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.348 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.348 DEBUG:boto:Method: GET 2024-03-04T16:33:50.348 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.348 DEBUG:boto:Data: 2024-03-04T16:33:50.348 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.348 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.349 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.349 DEBUG:boto:Params: {} 2024-03-04T16:33:50.349 DEBUG:boto:Token: None 2024-03-04T16:33:50.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df045f993432a41e-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.397 DEBUG:boto:Method: GET 2024-03-04T16:33:50.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.397 DEBUG:boto:Data: 2024-03-04T16:33:50.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.397 DEBUG:boto:Params: {} 2024-03-04T16:33:50.397 DEBUG:boto:Token: None 2024-03-04T16:33:50.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce970ddd03da012-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.444 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.444 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.444 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.445 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.445 DEBUG:boto:Method: GET 2024-03-04T16:33:50.445 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.445 DEBUG:boto:Data: 2024-03-04T16:33:50.445 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.445 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.445 DEBUG:boto:Params: {} 2024-03-04T16:33:50.445 DEBUG:boto:Token: None 2024-03-04T16:33:50.445 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f063c73e8bea011-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.448 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.449 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.449 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.449 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.449 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.449 DEBUG:boto:Method: GET 2024-03-04T16:33:50.449 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.449 DEBUG:boto:Data: 2024-03-04T16:33:50.450 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.450 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.450 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.450 DEBUG:boto:Params: {} 2024-03-04T16:33:50.450 DEBUG:boto:Token: None 2024-03-04T16:33:50.450 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7bfbdc5fb8c9b06-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.496 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.497 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.497 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.497 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.497 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.497 DEBUG:boto:Method: GET 2024-03-04T16:33:50.497 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.497 DEBUG:boto:Data: 2024-03-04T16:33:50.497 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.498 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.498 DEBUG:boto:Params: {} 2024-03-04T16:33:50.498 DEBUG:boto:Token: None 2024-03-04T16:33:50.498 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.498 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs= 2024-03-04T16:33:50.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs=', 'Content-Length': '0'} 2024-03-04T16:33:50.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075a19b8ab350d815-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.548 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.548 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.548 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.548 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.549 DEBUG:boto:Method: GET 2024-03-04T16:33:50.549 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.549 DEBUG:boto:Data: 2024-03-04T16:33:50.549 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.549 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.549 DEBUG:boto:Params: {} 2024-03-04T16:33:50.549 DEBUG:boto:Token: None 2024-03-04T16:33:50.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs= 2024-03-04T16:33:50.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs=', 'Content-Length': '0'} 2024-03-04T16:33:50.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000514eb8fac96ccd7-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.552 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.552 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.552 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.552 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.552 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.552 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.552 DEBUG:boto:Data: 2024-03-04T16:33:50.552 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.553 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.553 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.553 DEBUG:boto:Params: {} 2024-03-04T16:33:50.553 DEBUG:boto:Token: None 2024-03-04T16:33:50.553 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xI5kuqR6s1rUIAgcU5rpKWWgn5Q= 2024-03-04T16:33:50.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xI5kuqR6s1rUIAgcU5rpKWWgn5Q=', 'Content-Length': '0'} 2024-03-04T16:33:50.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087b8fe3365902c25-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.555 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.555 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.555 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.556 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.556 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.556 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.556 DEBUG:boto:Data: 2024-03-04T16:33:50.556 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.556 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.556 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.556 DEBUG:boto:Params: {} 2024-03-04T16:33:50.556 DEBUG:boto:Token: None 2024-03-04T16:33:50.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.556 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xI5kuqR6s1rUIAgcU5rpKWWgn5Q= 2024-03-04T16:33:50.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xI5kuqR6s1rUIAgcU5rpKWWgn5Q=', 'Content-Length': '0'} 2024-03-04T16:33:50.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002c8a902fc7da85f-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.559 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.559 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.559 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.559 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.559 DEBUG:boto:Method: GET 2024-03-04T16:33:50.559 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.559 DEBUG:boto:Data: 2024-03-04T16:33:50.560 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.560 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.560 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.560 DEBUG:boto:Params: {} 2024-03-04T16:33:50.560 DEBUG:boto:Token: None 2024-03-04T16:33:50.560 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.560 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs= 2024-03-04T16:33:50.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs=', 'Content-Length': '0'} 2024-03-04T16:33:50.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000409c367e1d0b834a-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.604 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.604 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.604 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.604 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.605 DEBUG:boto:Method: GET 2024-03-04T16:33:50.605 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.605 DEBUG:boto:Data: 2024-03-04T16:33:50.605 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.605 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.605 DEBUG:boto:Params: {} 2024-03-04T16:33:50.605 DEBUG:boto:Token: None 2024-03-04T16:33:50.605 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:50.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs= 2024-03-04T16:33:50.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:txpT5XfzPa6hy3FzAiikeLSXiSs=', 'Content-Length': '0'} 2024-03-04T16:33:50.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f3f31023e5881dd-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.648 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.648 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.648 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.649 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.649 DEBUG:boto:Method: GET 2024-03-04T16:33:50.649 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.649 DEBUG:boto:Data: 2024-03-04T16:33:50.649 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.649 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.649 DEBUG:boto:Params: {} 2024-03-04T16:33:50.649 DEBUG:boto:Token: None 2024-03-04T16:33:50.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb6da321e82eeefb-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.653 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.653 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.653 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.653 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.653 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.654 DEBUG:boto:Method: GET 2024-03-04T16:33:50.654 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.654 DEBUG:boto:Data: 2024-03-04T16:33:50.654 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.654 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.654 DEBUG:boto:Params: {} 2024-03-04T16:33:50.654 DEBUG:boto:Token: None 2024-03-04T16:33:50.654 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e9d94c9bcb123ea-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.657 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.658 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.658 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.658 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.658 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.658 DEBUG:boto:Data: 2024-03-04T16:33:50.658 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.658 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.658 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.658 DEBUG:boto:Params: {} 2024-03-04T16:33:50.658 DEBUG:boto:Token: None 2024-03-04T16:33:50.658 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.659 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed3e4f3876909551-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.662 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.662 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.662 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.662 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.663 DEBUG:boto:Data: 2024-03-04T16:33:50.663 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.663 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.663 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.663 DEBUG:boto:Params: {} 2024-03-04T16:33:50.663 DEBUG:boto:Token: None 2024-03-04T16:33:50.663 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.663 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5a85f7e2b8d242f-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.667 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.667 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.667 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.667 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.667 DEBUG:boto:Method: GET 2024-03-04T16:33:50.668 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.668 DEBUG:boto:Data: 2024-03-04T16:33:50.668 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.668 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.668 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.668 DEBUG:boto:Params: {} 2024-03-04T16:33:50.668 DEBUG:boto:Token: None 2024-03-04T16:33:50.668 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.668 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd8594f1c3d8059b-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.712 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.713 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.713 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.713 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.713 DEBUG:boto:Method: GET 2024-03-04T16:33:50.713 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.713 DEBUG:boto:Data: 2024-03-04T16:33:50.713 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.713 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.713 DEBUG:boto:Params: {} 2024-03-04T16:33:50.714 DEBUG:boto:Token: None 2024-03-04T16:33:50.714 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.714 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000225c047384e10a6d-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.760 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.760 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.760 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.760 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.761 DEBUG:boto:Method: GET 2024-03-04T16:33:50.761 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.761 DEBUG:boto:Data: 2024-03-04T16:33:50.761 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.761 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.761 DEBUG:boto:Params: {} 2024-03-04T16:33:50.761 DEBUG:boto:Token: None 2024-03-04T16:33:50.761 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.761 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e49eddb9634b72-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.764 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.765 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.765 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.765 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.765 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.765 DEBUG:boto:Method: GET 2024-03-04T16:33:50.765 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.765 DEBUG:boto:Data: 2024-03-04T16:33:50.765 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.766 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.766 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.766 DEBUG:boto:Params: {} 2024-03-04T16:33:50.766 DEBUG:boto:Token: None 2024-03-04T16:33:50.766 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e714366e23cea4d-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.769 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.770 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.770 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.771 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.771 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.771 DEBUG:boto:Method: GET 2024-03-04T16:33:50.771 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.771 DEBUG:boto:Data: 2024-03-04T16:33:50.771 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.771 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.771 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.771 DEBUG:boto:Params: {} 2024-03-04T16:33:50.771 DEBUG:boto:Token: None 2024-03-04T16:33:50.771 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.772 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU= 2024-03-04T16:33:50.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU=', 'Content-Length': '0'} 2024-03-04T16:33:50.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df484339854d9249-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.816 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.816 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.816 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.817 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.817 DEBUG:boto:Method: GET 2024-03-04T16:33:50.817 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.817 DEBUG:boto:Data: 2024-03-04T16:33:50.817 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.817 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.817 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.817 DEBUG:boto:Params: {} 2024-03-04T16:33:50.817 DEBUG:boto:Token: None 2024-03-04T16:33:50.818 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.818 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU= 2024-03-04T16:33:50.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU=', 'Content-Length': '0'} 2024-03-04T16:33:50.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fedbb18a176a519-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.820 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.820 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.820 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.821 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.821 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.821 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.821 DEBUG:boto:Data: 2024-03-04T16:33:50.821 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.821 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.821 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.821 DEBUG:boto:Params: {} 2024-03-04T16:33:50.821 DEBUG:boto:Token: None 2024-03-04T16:33:50.821 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.822 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GfgikKfHbnS9CwMImbY7cFL1RBc= 2024-03-04T16:33:50.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GfgikKfHbnS9CwMImbY7cFL1RBc=', 'Content-Length': '0'} 2024-03-04T16:33:50.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000674a406dda705a57-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.824 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.824 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.824 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.824 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.824 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.824 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.824 DEBUG:boto:Data: 2024-03-04T16:33:50.824 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.825 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.825 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.825 DEBUG:boto:Params: {} 2024-03-04T16:33:50.825 DEBUG:boto:Token: None 2024-03-04T16:33:50.825 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.825 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GfgikKfHbnS9CwMImbY7cFL1RBc= 2024-03-04T16:33:50.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GfgikKfHbnS9CwMImbY7cFL1RBc=', 'Content-Length': '0'} 2024-03-04T16:33:50.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6085a1b248948d5-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.827 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.827 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.828 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.828 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.828 DEBUG:boto:Method: GET 2024-03-04T16:33:50.828 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.828 DEBUG:boto:Data: 2024-03-04T16:33:50.828 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.828 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.828 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.828 DEBUG:boto:Params: {} 2024-03-04T16:33:50.828 DEBUG:boto:Token: None 2024-03-04T16:33:50.828 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU= 2024-03-04T16:33:50.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU=', 'Content-Length': '0'} 2024-03-04T16:33:50.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000355b1b250b3a81b8-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.872 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.872 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.873 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.873 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.873 DEBUG:boto:Method: GET 2024-03-04T16:33:50.873 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.873 DEBUG:boto:Data: 2024-03-04T16:33:50.873 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.873 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.873 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.873 DEBUG:boto:Params: {} 2024-03-04T16:33:50.873 DEBUG:boto:Token: None 2024-03-04T16:33:50.874 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:50.874 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU= 2024-03-04T16:33:50.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bs4F5xQsvtt1Q+Vu3Ae+zzbANQU=', 'Content-Length': '0'} 2024-03-04T16:33:50.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000446c3dae2adb2b13-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.877 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.877 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.877 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.877 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.877 DEBUG:boto:Method: GET 2024-03-04T16:33:50.877 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.877 DEBUG:boto:Data: 2024-03-04T16:33:50.877 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.877 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.877 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.877 DEBUG:boto:Params: {} 2024-03-04T16:33:50.878 DEBUG:boto:Token: None 2024-03-04T16:33:50.878 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.878 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001fa0dc88755d922-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.928 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.929 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:50.929 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:50.929 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.929 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:50.929 DEBUG:boto:Method: GET 2024-03-04T16:33:50.929 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:50.929 DEBUG:boto:Data: 2024-03-04T16:33:50.929 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.929 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.929 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.930 DEBUG:boto:Params: {} 2024-03-04T16:33:50.930 DEBUG:boto:Token: None 2024-03-04T16:33:50.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/ 2024-03-04T16:33:50.930 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0= 2024-03-04T16:33:50.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rqewZGKe3VsewUO/cszM3p8OHN0=', 'Content-Length': '0'} 2024-03-04T16:33:50.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5201db12aa0423-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.934 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:50.934 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.934 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.934 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.934 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.935 DEBUG:boto:Data: 2024-03-04T16:33:50.935 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.935 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.935 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.935 DEBUG:boto:Params: {} 2024-03-04T16:33:50.935 DEBUG:boto:Token: None 2024-03-04T16:33:50.935 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.935 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1d5deac58c766f7-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.939 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.939 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.939 DEBUG:boto:Method: HEAD 2024-03-04T16:33:50.940 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:50.940 DEBUG:boto:Data: 2024-03-04T16:33:50.940 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.940 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.940 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.940 DEBUG:boto:Params: {} 2024-03-04T16:33:50.940 DEBUG:boto:Token: None 2024-03-04T16:33:50.940 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj 2024-03-04T16:33:50.940 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys= 2024-03-04T16:33:50.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xQForOs6edkhURtOxxQfLmHUUys=', 'Content-Length': '0'} 2024-03-04T16:33:50.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8f184fdace18861-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:50.945 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.945 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.945 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.945 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.945 DEBUG:boto:Method: GET 2024-03-04T16:33:50.945 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.945 DEBUG:boto:Data: 2024-03-04T16:33:50.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:50.946 DEBUG:boto:Port: 8001 2024-03-04T16:33:50.946 DEBUG:boto:Params: {} 2024-03-04T16:33:50.946 DEBUG:boto:Token: None 2024-03-04T16:33:50.946 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1712b6ab3f1d613-0065e5f7ee-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:50.988 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:50.988 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:50.988 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.989 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.989 DEBUG:boto:Method: GET 2024-03-04T16:33:50.989 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.989 DEBUG:boto:Data: 2024-03-04T16:33:50.989 DEBUG:boto:Headers: {} 2024-03-04T16:33:50.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:50.989 DEBUG:boto:Port: 8000 2024-03-04T16:33:50.989 DEBUG:boto:Params: {} 2024-03-04T16:33:50.989 DEBUG:boto:Token: None 2024-03-04T16:33:50.989 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:50 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:50.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y= 2024-03-04T16:33:50.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l9APGaqYpvapxI5cGKUGmEztQ7Y=', 'Content-Length': '0'} 2024-03-04T16:33:50.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f032e1f6785a3d45-0065e5f7ee-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.032 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.032 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.032 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.033 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.033 DEBUG:boto:Method: GET 2024-03-04T16:33:51.033 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.033 DEBUG:boto:Data: 2024-03-04T16:33:51.033 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.033 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.033 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.033 DEBUG:boto:Params: {} 2024-03-04T16:33:51.033 DEBUG:boto:Token: None 2024-03-04T16:33:51.033 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003208ad82674ca5d7-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.037 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.037 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.037 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.037 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.037 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.038 DEBUG:boto:Method: GET 2024-03-04T16:33:51.038 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.038 DEBUG:boto:Data: 2024-03-04T16:33:51.038 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.038 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.038 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.038 DEBUG:boto:Params: {} 2024-03-04T16:33:51.038 DEBUG:boto:Token: None 2024-03-04T16:33:51.038 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8b0593700bccaa-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.042 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.043 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.043 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.043 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.043 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.043 DEBUG:boto:Method: GET 2024-03-04T16:33:51.043 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.043 DEBUG:boto:Data: 2024-03-04T16:33:51.043 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.043 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.044 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.044 DEBUG:boto:Params: {} 2024-03-04T16:33:51.044 DEBUG:boto:Token: None 2024-03-04T16:33:51.044 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.044 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4= 2024-03-04T16:33:51.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4=', 'Content-Length': '0'} 2024-03-04T16:33:51.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7371e5e2a7bebf0-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.088 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.088 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.088 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.089 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.089 DEBUG:boto:Method: GET 2024-03-04T16:33:51.089 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.089 DEBUG:boto:Data: 2024-03-04T16:33:51.089 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.089 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.089 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.089 DEBUG:boto:Params: {} 2024-03-04T16:33:51.089 DEBUG:boto:Token: None 2024-03-04T16:33:51.090 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.090 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4= 2024-03-04T16:33:51.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4=', 'Content-Length': '0'} 2024-03-04T16:33:51.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027ffdb4f7d2fbfce-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.092 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.092 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.092 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.093 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.093 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.093 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.093 DEBUG:boto:Data: 2024-03-04T16:33:51.093 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.093 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.093 DEBUG:boto:Params: {} 2024-03-04T16:33:51.094 DEBUG:boto:Token: None 2024-03-04T16:33:51.094 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.094 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PH7LbbvWYWzOP69iKsZEpbwdtu4= 2024-03-04T16:33:51.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PH7LbbvWYWzOP69iKsZEpbwdtu4=', 'Content-Length': '0'} 2024-03-04T16:33:51.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd2cf0e82e487289-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.097 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.097 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.097 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.098 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.098 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.098 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.098 DEBUG:boto:Data: 2024-03-04T16:33:51.098 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.098 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.099 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.099 DEBUG:boto:Params: {} 2024-03-04T16:33:51.099 DEBUG:boto:Token: None 2024-03-04T16:33:51.099 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.099 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PH7LbbvWYWzOP69iKsZEpbwdtu4= 2024-03-04T16:33:51.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PH7LbbvWYWzOP69iKsZEpbwdtu4=', 'Content-Length': '0'} 2024-03-04T16:33:51.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb64fd53d8d70a48-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.102 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.102 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.102 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.102 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.102 DEBUG:boto:Method: GET 2024-03-04T16:33:51.102 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.102 DEBUG:boto:Data: 2024-03-04T16:33:51.102 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.102 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.102 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.103 DEBUG:boto:Params: {} 2024-03-04T16:33:51.103 DEBUG:boto:Token: None 2024-03-04T16:33:51.103 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.103 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4= 2024-03-04T16:33:51.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4=', 'Content-Length': '0'} 2024-03-04T16:33:51.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007c6ba6af0d6ed8e-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.105 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.106 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.106 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.106 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.106 DEBUG:boto:Method: GET 2024-03-04T16:33:51.106 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.106 DEBUG:boto:Data: 2024-03-04T16:33:51.106 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.106 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.106 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.106 DEBUG:boto:Params: {} 2024-03-04T16:33:51.107 DEBUG:boto:Token: None 2024-03-04T16:33:51.107 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:51.107 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4= 2024-03-04T16:33:51.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UEbcIoqC9Ovlg2KaqU8h6p0u+I4=', 'Content-Length': '0'} 2024-03-04T16:33:51.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8ad99f8d7fc09a9-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.152 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.152 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.152 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.153 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.153 DEBUG:boto:Method: GET 2024-03-04T16:33:51.153 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.153 DEBUG:boto:Data: 2024-03-04T16:33:51.153 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.153 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.153 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.153 DEBUG:boto:Params: {} 2024-03-04T16:33:51.153 DEBUG:boto:Token: None 2024-03-04T16:33:51.153 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048797e141ea3feb7-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.156 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.157 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.157 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.157 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.157 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.157 DEBUG:boto:Method: GET 2024-03-04T16:33:51.157 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.157 DEBUG:boto:Data: 2024-03-04T16:33:51.157 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.158 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.158 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.158 DEBUG:boto:Params: {} 2024-03-04T16:33:51.158 DEBUG:boto:Token: None 2024-03-04T16:33:51.158 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.158 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5001f65e21d0c55-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.208 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.209 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.209 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.209 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.209 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.209 DEBUG:boto:Data: 2024-03-04T16:33:51.209 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.209 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.209 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.209 DEBUG:boto:Params: {} 2024-03-04T16:33:51.209 DEBUG:boto:Token: None 2024-03-04T16:33:51.210 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.210 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011967724947671bd-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.214 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.214 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.214 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.214 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.214 DEBUG:boto:Data: 2024-03-04T16:33:51.214 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.214 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.214 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.214 DEBUG:boto:Params: {} 2024-03-04T16:33:51.214 DEBUG:boto:Token: None 2024-03-04T16:33:51.215 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.215 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d911bcd2d2c3fd23-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.218 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.218 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.219 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.219 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.219 DEBUG:boto:Method: GET 2024-03-04T16:33:51.219 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.219 DEBUG:boto:Data: 2024-03-04T16:33:51.219 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.219 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.219 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.219 DEBUG:boto:Params: {} 2024-03-04T16:33:51.219 DEBUG:boto:Token: None 2024-03-04T16:33:51.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.220 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d2f5074a5db8d98-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.264 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.264 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.264 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.264 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.265 DEBUG:boto:Method: GET 2024-03-04T16:33:51.265 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.265 DEBUG:boto:Data: 2024-03-04T16:33:51.265 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.265 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.265 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.265 DEBUG:boto:Params: {} 2024-03-04T16:33:51.265 DEBUG:boto:Token: None 2024-03-04T16:33:51.265 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d068071f79d3f70-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.308 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.308 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.308 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.309 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.309 DEBUG:boto:Method: GET 2024-03-04T16:33:51.309 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.309 DEBUG:boto:Data: 2024-03-04T16:33:51.309 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.309 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.309 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.309 DEBUG:boto:Params: {} 2024-03-04T16:33:51.309 DEBUG:boto:Token: None 2024-03-04T16:33:51.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2c69a2d4e7bf4c-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.313 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.314 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.314 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.314 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.314 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.314 DEBUG:boto:Method: GET 2024-03-04T16:33:51.314 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.314 DEBUG:boto:Data: 2024-03-04T16:33:51.314 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.314 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:33:51.314 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.315 DEBUG:boto:Params: {} 2024-03-04T16:33:51.315 DEBUG:boto:Token: None 2024-03-04T16:33:51.315 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.315 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187920808ea58165-0065e5f7ef-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.364 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone3, test-zone1} 2024-03-04T16:33:51.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-22 zones={test-zone3, test-zone2} 2024-03-04T16:33:51.365 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.365 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.365 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.365 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:51.365 DEBUG:boto:Data: 2024-03-04T16:33:51.365 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.365 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.366 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.366 DEBUG:boto:Params: {} 2024-03-04T16:33:51.366 DEBUG:boto:Token: None 2024-03-04T16:33:51.366 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.366 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fM9Trf+ZaCF4EB9Juthf7Chb7R4= 2024-03-04T16:33:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fM9Trf+ZaCF4EB9Juthf7Chb7R4=', 'Content-Length': '0'} 2024-03-04T16:33:51.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e325a609ecdc965f-0065e5f7ef-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.369 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.369 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.369 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.369 DEBUG:boto:Path: /sbnvju-22/ 2024-03-04T16:33:51.369 DEBUG:boto:Data: 2024-03-04T16:33:51.369 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.370 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.370 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.370 DEBUG:boto:Params: {} 2024-03-04T16:33:51.370 DEBUG:boto:Token: None 2024-03-04T16:33:51.370 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.370 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fM9Trf+ZaCF4EB9Juthf7Chb7R4= 2024-03-04T16:33:51.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fM9Trf+ZaCF4EB9Juthf7Chb7R4=', 'Content-Length': '0'} 2024-03-04T16:33:51.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b1dddc27f1ceec6-0065e5f7ef-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:33:51.373 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.373 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.373 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.374 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.374 DEBUG:boto:Method: GET 2024-03-04T16:33:51.374 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.374 DEBUG:boto:Data: 2024-03-04T16:33:51.374 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.374 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.374 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.374 DEBUG:boto:Params: {} 2024-03-04T16:33:51.374 DEBUG:boto:Token: None 2024-03-04T16:33:51.374 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/?versions 2024-03-04T16:33:51.375 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4= 2024-03-04T16:33:51.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4=', 'Content-Length': '0'} 2024-03-04T16:33:51.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003636f60acbcb7b16-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.379 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.380 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:51.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:33:51.383 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.383 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.383 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.384 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.384 DEBUG:boto:Method: GET 2024-03-04T16:33:51.384 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.384 DEBUG:boto:Data: 2024-03-04T16:33:51.384 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.384 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.384 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.384 DEBUG:boto:Params: {} 2024-03-04T16:33:51.384 DEBUG:boto:Token: None 2024-03-04T16:33:51.385 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/?versions 2024-03-04T16:33:51.385 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4= 2024-03-04T16:33:51.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4=', 'Content-Length': '0'} 2024-03-04T16:33:51.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000699242e75c8581f3-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.389 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-03-04T16:33:51.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:33:51.393 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.393 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.393 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.393 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.393 DEBUG:boto:Method: GET 2024-03-04T16:33:51.393 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.393 DEBUG:boto:Data: 2024-03-04T16:33:51.393 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.394 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.394 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.394 DEBUG:boto:Params: {} 2024-03-04T16:33:51.394 DEBUG:boto:Token: None 2024-03-04T16:33:51.394 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/?versions 2024-03-04T16:33:51.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4= 2024-03-04T16:33:51.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4=', 'Content-Length': '0'} 2024-03-04T16:33:51.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004877ed0150f35637-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.398 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.399 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.399 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.399 DEBUG:boto:path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.400 DEBUG:boto:auth_path=/sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.400 DEBUG:boto:Method: GET 2024-03-04T16:33:51.400 DEBUG:boto:Path: /sbnvju-22/?versions&max-keys=999 2024-03-04T16:33:51.400 DEBUG:boto:Data: 2024-03-04T16:33:51.400 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.400 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.400 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.400 DEBUG:boto:Params: {} 2024-03-04T16:33:51.400 DEBUG:boto:Token: None 2024-03-04T16:33:51.400 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/?versions 2024-03-04T16:33:51.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4= 2024-03-04T16:33:51.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:D+QhwbU3q3+84VZBvVV6zn8NEs4=', 'Content-Length': '0'} 2024-03-04T16:33:51.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe83bb05facbf8ef-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.404 DEBUG:boto:b'sbnvju-22999falsedummytoPqckOG.fgRlKyYIVzHmRc0d5Gw9KItrue2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7OSlhMe-S45YQ-C5o3I-Cprru4mMZYitrue2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhfXmZko7NeFEk5eOt6OzuIOcw0ti-Ktfalse2024-03-04T16:29:21.456Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobjybm.zcpg4NCgTW0qlMotVwsu1Qapfalse2024-03-04T16:29:21.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj01uuPC1IwgjYFQiRu10-OMZvc7W7i9dfalse2024-03-04T16:29:21.410Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji4ms9tGVGjdbOyxew4b4Yd0zBU8rzCbfalse2024-03-04T16:29:21.300Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt8CpszqgdrbNp1tmFmKatQEXeCtSLVgfalse2024-03-04T16:29:21.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXUAluvGVKdo-73mqoSBEEUwxIHvrmM4false2024-03-04T16:29:21.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobja2H67AHZ7-ZbPRGTiQGJV-y.JdDILOtfalse2024-03-04T16:29:21.286Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgNT65zfABYfwW89PXHXbF.jE4EXc0Wlfalse2024-03-04T16:29:21.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj04Bk218QitWUyZShNL.038YuwPvLhqlfalse2024-03-04T16:29:21.200Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBLi4aTmZK0uBvmzaBcObjwKVJqxtERdfalse2024-03-04T16:29:21.140Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtWwOAgDLiLza3F6cTWddXwRcOE7WT1Bfalse2024-03-04T16:29:21.125Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGvfalse2024-03-04T16:29:21.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBiNnWumILmRGRqS4nbXjGGtEQEPDuu2false2024-03-04T16:29:21.107Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH8LlsoTnXLqPNb1-opllLInHMzeoYsQfalse2024-03-04T16:29:21.041Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gSfalse2024-03-04T16:29:20.996Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXiCW4ml6JGIKsAt..485xm6JEN33ZeJfalse2024-03-04T16:29:21.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjf9y2fO3FZIWX9Lki2lP3yXMrAmJcFhkfalse2024-03-04T16:29:21.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUWMoj2KG9hGrAO1d83MedK1E8JGdxkKfalse2024-03-04T16:29:20.949Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ1-iuDFIejtz7F1ccZKbN2mDV7h.h9Rfalse2024-03-04T16:29:20.913Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjahIF6D4RiReN93jnXI2J7WYQmcubz3Lfalse2024-03-04T16:29:20.928Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8false2024-03-04T16:29:20.898Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRP2aX6PN4tqCxXe-c6sZV5.fhRwjvzHfalse2024-03-04T16:29:20.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWqFQw08WT4szpVVRJMZzxcn8IIQaJrXfalse2024-03-04T16:29:20.805Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60c45DIWYEmYXg2q5fpDBATrHDwl4Gpfalse2024-03-04T16:29:20.674Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjY9VfGbVuWMLRnERQj5tDOof79HPhWHlfalse2024-03-04T16:29:20.690Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjus1Q-hNLD9lb1a0qvK1BMsAcZBvQaaNfalse2024-03-04T16:29:20.743Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1slxKjMQBTNKx8wPzjpbP7zo48qatQbfalse2024-03-04T16:29:20.639Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHclfl8Hokp6qeo82Mt5xCpcu3CPrjsSfalse2024-03-04T16:29:20.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFHfalse2024-03-04T16:29:20.657Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:51.406 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.406 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.406 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.406 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.406 DEBUG:boto:Method: GET 2024-03-04T16:33:51.406 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.406 DEBUG:boto:Data: 2024-03-04T16:33:51.406 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.406 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.407 DEBUG:boto:Params: {} 2024-03-04T16:33:51.407 DEBUG:boto:Token: None 2024-03-04T16:33:51.407 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.407 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3312268f0d27258-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.409 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.409 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.409 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.409 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.409 DEBUG:boto:Method: GET 2024-03-04T16:33:51.409 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.410 DEBUG:boto:Data: 2024-03-04T16:33:51.410 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.410 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.410 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.410 DEBUG:boto:Params: {} 2024-03-04T16:33:51.410 DEBUG:boto:Token: None 2024-03-04T16:33:51.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a71975c454c8033-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.412 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.412 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.412 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.412 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.413 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.413 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.413 DEBUG:boto:Data: 2024-03-04T16:33:51.413 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.413 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.413 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.413 DEBUG:boto:Params: {} 2024-03-04T16:33:51.413 DEBUG:boto:Token: None 2024-03-04T16:33:51.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:X4sZuBED1cbe4Z35jHlwI0Xaph8= 2024-03-04T16:33:51.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:X4sZuBED1cbe4Z35jHlwI0Xaph8=', 'Content-Length': '0'} 2024-03-04T16:33:51.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003c6a85d3f948767-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.415 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.416 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.416 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.416 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.416 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.416 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.416 DEBUG:boto:Data: 2024-03-04T16:33:51.416 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.416 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.416 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.416 DEBUG:boto:Params: {} 2024-03-04T16:33:51.416 DEBUG:boto:Token: None 2024-03-04T16:33:51.417 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:X4sZuBED1cbe4Z35jHlwI0Xaph8= 2024-03-04T16:33:51.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:X4sZuBED1cbe4Z35jHlwI0Xaph8=', 'Content-Length': '0'} 2024-03-04T16:33:51.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9dd015a71971335-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:51.419 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.419 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.419 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.419 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.419 DEBUG:boto:Method: GET 2024-03-04T16:33:51.419 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.419 DEBUG:boto:Data: 2024-03-04T16:33:51.419 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.420 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.420 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.420 DEBUG:boto:Params: {} 2024-03-04T16:33:51.420 DEBUG:boto:Token: None 2024-03-04T16:33:51.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.420 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017cbf2543df5a3fc-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.422 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.422 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.423 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.423 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.423 DEBUG:boto:Method: GET 2024-03-04T16:33:51.423 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.423 DEBUG:boto:Data: 2024-03-04T16:33:51.423 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.423 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.423 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.423 DEBUG:boto:Params: {} 2024-03-04T16:33:51.423 DEBUG:boto:Token: None 2024-03-04T16:33:51.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.424 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cd0ea3f9c4490d6-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.426 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.426 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.426 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.426 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.426 DEBUG:boto:Method: GET 2024-03-04T16:33:51.426 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.426 DEBUG:boto:Data: 2024-03-04T16:33:51.426 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.426 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.426 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.426 DEBUG:boto:Params: {} 2024-03-04T16:33:51.426 DEBUG:boto:Token: None 2024-03-04T16:33:51.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.427 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032548f99d1ee0770-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.472 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.473 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.473 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.473 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.473 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.473 DEBUG:boto:Method: GET 2024-03-04T16:33:51.473 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.473 DEBUG:boto:Data: 2024-03-04T16:33:51.473 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.473 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.473 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.473 DEBUG:boto:Params: {} 2024-03-04T16:33:51.474 DEBUG:boto:Token: None 2024-03-04T16:33:51.474 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a4de9b4b93aa50-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.477 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.477 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.477 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.477 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.477 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:51.477 DEBUG:boto:Data: 2024-03-04T16:33:51.478 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.478 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.478 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.478 DEBUG:boto:Params: {} 2024-03-04T16:33:51.478 DEBUG:boto:Token: None 2024-03-04T16:33:51.478 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy 2024-03-04T16:33:51.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hWABp9VkMkNGcMa5YYWzAOy6Uo0= 2024-03-04T16:33:51.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hWABp9VkMkNGcMa5YYWzAOy6Uo0=', 'Content-Length': '0'} 2024-03-04T16:33:51.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050e02ff419591215-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.481 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.481 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.481 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.481 DEBUG:boto:Path: /sbnvju-22/dummy 2024-03-04T16:33:51.481 DEBUG:boto:Data: 2024-03-04T16:33:51.481 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.481 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.481 DEBUG:boto:Params: {} 2024-03-04T16:33:51.481 DEBUG:boto:Token: None 2024-03-04T16:33:51.482 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy 2024-03-04T16:33:51.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hWABp9VkMkNGcMa5YYWzAOy6Uo0= 2024-03-04T16:33:51.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hWABp9VkMkNGcMa5YYWzAOy6Uo0=', 'Content-Length': '0'} 2024-03-04T16:33:51.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adee2ac5a6be2d2f-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-03-04T16:33:51.484 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.484 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.484 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.484 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.484 DEBUG:boto:Method: GET 2024-03-04T16:33:51.485 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.485 DEBUG:boto:Data: 2024-03-04T16:33:51.485 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.485 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.485 DEBUG:boto:Params: {} 2024-03-04T16:33:51.485 DEBUG:boto:Token: None 2024-03-04T16:33:51.485 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e8c82703b5fb431-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.488 DEBUG:boto:path=/sbnvju-22/dummy 2024-03-04T16:33:51.488 DEBUG:boto:auth_path=/sbnvju-22/dummy 2024-03-04T16:33:51.488 DEBUG:boto:path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.489 DEBUG:boto:auth_path=/sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.489 DEBUG:boto:Method: GET 2024-03-04T16:33:51.489 DEBUG:boto:Path: /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.489 DEBUG:boto:Data: 2024-03-04T16:33:51.489 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.489 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.489 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.489 DEBUG:boto:Params: {} 2024-03-04T16:33:51.489 DEBUG:boto:Token: None 2024-03-04T16:33:51.490 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/dummy?versionId=toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI 2024-03-04T16:33:51.490 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY= 2024-03-04T16:33:51.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B2pkOz6+M/AKzVG1wwhvHPCClGY=', 'Content-Length': '0'} 2024-03-04T16:33:51.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:28:19 GMT'), ('x-amz-version-id', 'toPqckOG.fgRlKyYIVzHmRc0d5Gw9KI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fefd66f5e0f37ccc-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.492 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.492 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.492 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.492 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.492 DEBUG:boto:Method: GET 2024-03-04T16:33:51.492 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.492 DEBUG:boto:Data: 2024-03-04T16:33:51.492 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.493 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.493 DEBUG:boto:Params: {} 2024-03-04T16:33:51.493 DEBUG:boto:Token: None 2024-03-04T16:33:51.493 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.493 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b3b266a23c2fe76-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.540 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.540 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.541 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.541 DEBUG:boto:path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.541 DEBUG:boto:auth_path=/sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.541 DEBUG:boto:Method: GET 2024-03-04T16:33:51.541 DEBUG:boto:Path: /sbnvju-22/?prefix=dummy 2024-03-04T16:33:51.541 DEBUG:boto:Data: 2024-03-04T16:33:51.541 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.541 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.541 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.541 DEBUG:boto:Params: {} 2024-03-04T16:33:51.542 DEBUG:boto:Token: None 2024-03-04T16:33:51.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250f66e5181b9681-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.544 DEBUG:boto:b'sbnvju-22dummy1000falsedummy2024-03-04T16:28:19.093Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.545 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.545 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.545 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.545 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.545 DEBUG:boto:Method: GET 2024-03-04T16:33:51.545 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.545 DEBUG:boto:Data: 2024-03-04T16:33:51.545 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.546 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.546 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.546 DEBUG:boto:Params: {} 2024-03-04T16:33:51.546 DEBUG:boto:Token: None 2024-03-04T16:33:51.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e229a686b044b07f-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.592 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.592 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.592 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.592 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.592 DEBUG:boto:Method: GET 2024-03-04T16:33:51.593 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.593 DEBUG:boto:Data: 2024-03-04T16:33:51.593 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.593 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.593 DEBUG:boto:Params: {} 2024-03-04T16:33:51.593 DEBUG:boto:Token: None 2024-03-04T16:33:51.593 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.593 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e90b77983b9ef46-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.596 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.596 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.596 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.596 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.596 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.597 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.597 DEBUG:boto:Data: 2024-03-04T16:33:51.597 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.597 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.597 DEBUG:boto:Params: {} 2024-03-04T16:33:51.597 DEBUG:boto:Token: None 2024-03-04T16:33:51.597 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/0jz9+q/wM4DpgqB6x54vNhJ8Y= 2024-03-04T16:33:51.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/0jz9+q/wM4DpgqB6x54vNhJ8Y=', 'Content-Length': '0'} 2024-03-04T16:33:51.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01cc479c0de4d2a-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.599 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.599 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.600 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.600 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.600 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.600 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.600 DEBUG:boto:Data: 2024-03-04T16:33:51.600 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.600 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.600 DEBUG:boto:Params: {} 2024-03-04T16:33:51.600 DEBUG:boto:Token: None 2024-03-04T16:33:51.601 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i/0jz9+q/wM4DpgqB6x54vNhJ8Y= 2024-03-04T16:33:51.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i/0jz9+q/wM4DpgqB6x54vNhJ8Y=', 'Content-Length': '0'} 2024-03-04T16:33:51.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ccadfd4632f3294-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.603 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.603 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.603 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.603 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.603 DEBUG:boto:Method: GET 2024-03-04T16:33:51.604 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.604 DEBUG:boto:Data: 2024-03-04T16:33:51.604 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.604 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.604 DEBUG:boto:Params: {} 2024-03-04T16:33:51.604 DEBUG:boto:Token: None 2024-03-04T16:33:51.604 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.604 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed8a658ffbfdd16a-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.648 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.648 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.648 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.648 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.649 DEBUG:boto:Method: GET 2024-03-04T16:33:51.649 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.649 DEBUG:boto:Data: 2024-03-04T16:33:51.649 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.649 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.649 DEBUG:boto:Params: {} 2024-03-04T16:33:51.649 DEBUG:boto:Token: None 2024-03-04T16:33:51.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009504c58f847bee60-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.652 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.652 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.652 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.652 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.652 DEBUG:boto:Method: GET 2024-03-04T16:33:51.652 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.652 DEBUG:boto:Data: 2024-03-04T16:33:51.652 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.653 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.653 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.653 DEBUG:boto:Params: {} 2024-03-04T16:33:51.653 DEBUG:boto:Token: None 2024-03-04T16:33:51.653 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.653 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48a565e5a32cdba-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.696 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.697 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.697 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.697 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.697 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.697 DEBUG:boto:Method: GET 2024-03-04T16:33:51.697 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.697 DEBUG:boto:Data: 2024-03-04T16:33:51.697 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.697 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.697 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.697 DEBUG:boto:Params: {} 2024-03-04T16:33:51.698 DEBUG:boto:Token: None 2024-03-04T16:33:51.698 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9fbc078d68489a-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.701 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.701 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.701 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.701 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.701 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.702 DEBUG:boto:Data: 2024-03-04T16:33:51.702 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.702 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.702 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.702 DEBUG:boto:Params: {} 2024-03-04T16:33:51.702 DEBUG:boto:Token: None 2024-03-04T16:33:51.702 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.702 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045031a14893f54cf-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.706 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.706 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.706 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.707 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.707 DEBUG:boto:Data: 2024-03-04T16:33:51.707 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.707 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.707 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.707 DEBUG:boto:Params: {} 2024-03-04T16:33:51.707 DEBUG:boto:Token: None 2024-03-04T16:33:51.707 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.707 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bf6360eb2d6c2ac-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.711 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.711 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.711 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.711 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.711 DEBUG:boto:Method: GET 2024-03-04T16:33:51.711 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.712 DEBUG:boto:Data: 2024-03-04T16:33:51.712 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.712 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.712 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.712 DEBUG:boto:Params: {} 2024-03-04T16:33:51.712 DEBUG:boto:Token: None 2024-03-04T16:33:51.712 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.712 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec173aeba1e14ee0-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.756 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.756 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.756 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.756 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.757 DEBUG:boto:Method: GET 2024-03-04T16:33:51.757 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.757 DEBUG:boto:Data: 2024-03-04T16:33:51.757 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.757 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.757 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.757 DEBUG:boto:Params: {} 2024-03-04T16:33:51.757 DEBUG:boto:Token: None 2024-03-04T16:33:51.757 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.758 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e33894d1a285f443-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.760 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.760 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.760 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.760 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.760 DEBUG:boto:Method: GET 2024-03-04T16:33:51.760 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.761 DEBUG:boto:Data: 2024-03-04T16:33:51.761 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.761 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.761 DEBUG:boto:Params: {} 2024-03-04T16:33:51.761 DEBUG:boto:Token: None 2024-03-04T16:33:51.761 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.761 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f67bd57c9e0e3a73-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.812 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.813 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.813 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.813 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.813 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.813 DEBUG:boto:Method: GET 2024-03-04T16:33:51.813 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.813 DEBUG:boto:Data: 2024-03-04T16:33:51.813 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.814 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.814 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.814 DEBUG:boto:Params: {} 2024-03-04T16:33:51.814 DEBUG:boto:Token: None 2024-03-04T16:33:51.814 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.814 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078962a63b31b30d7-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.817 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.818 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.818 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.818 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.818 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.818 DEBUG:boto:Method: GET 2024-03-04T16:33:51.818 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.818 DEBUG:boto:Data: 2024-03-04T16:33:51.818 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.818 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.818 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.818 DEBUG:boto:Params: {} 2024-03-04T16:33:51.819 DEBUG:boto:Token: None 2024-03-04T16:33:51.819 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.819 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k= 2024-03-04T16:33:51.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k=', 'Content-Length': '0'} 2024-03-04T16:33:51.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090bb654cbb03cd9c-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.821 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.821 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.821 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.821 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.821 DEBUG:boto:Method: GET 2024-03-04T16:33:51.822 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.822 DEBUG:boto:Data: 2024-03-04T16:33:51.822 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.822 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.822 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.822 DEBUG:boto:Params: {} 2024-03-04T16:33:51.822 DEBUG:boto:Token: None 2024-03-04T16:33:51.823 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.823 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k= 2024-03-04T16:33:51.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k=', 'Content-Length': '0'} 2024-03-04T16:33:51.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69033ce10342bff-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.825 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.825 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.826 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.826 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.826 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.826 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.826 DEBUG:boto:Data: 2024-03-04T16:33:51.826 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.826 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.826 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.826 DEBUG:boto:Params: {} 2024-03-04T16:33:51.826 DEBUG:boto:Token: None 2024-03-04T16:33:51.826 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.827 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B6EStEzhVlBC9/imYQk5uDe7Ct4= 2024-03-04T16:33:51.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B6EStEzhVlBC9/imYQk5uDe7Ct4=', 'Content-Length': '0'} 2024-03-04T16:33:51.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df7319ac7d6c270d-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.829 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.829 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.829 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.829 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.829 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.829 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.829 DEBUG:boto:Data: 2024-03-04T16:33:51.830 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.830 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.830 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.830 DEBUG:boto:Params: {} 2024-03-04T16:33:51.830 DEBUG:boto:Token: None 2024-03-04T16:33:51.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:B6EStEzhVlBC9/imYQk5uDe7Ct4= 2024-03-04T16:33:51.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:B6EStEzhVlBC9/imYQk5uDe7Ct4=', 'Content-Length': '0'} 2024-03-04T16:33:51.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063ebb9fc8251a739-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.833 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.833 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.833 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.833 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.833 DEBUG:boto:Method: GET 2024-03-04T16:33:51.833 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.834 DEBUG:boto:Data: 2024-03-04T16:33:51.834 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.834 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.834 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.834 DEBUG:boto:Params: {} 2024-03-04T16:33:51.834 DEBUG:boto:Token: None 2024-03-04T16:33:51.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.834 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k= 2024-03-04T16:33:51.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k=', 'Content-Length': '0'} 2024-03-04T16:33:51.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e17cf94a21d09a7-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.880 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.880 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.880 DEBUG:boto:path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.880 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.880 DEBUG:boto:Method: GET 2024-03-04T16:33:51.881 DEBUG:boto:Path: /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.881 DEBUG:boto:Data: 2024-03-04T16:33:51.881 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.881 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.881 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.881 DEBUG:boto:Params: {} 2024-03-04T16:33:51.881 DEBUG:boto:Token: None 2024-03-04T16:33:51.881 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt 2024-03-04T16:33:51.881 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k= 2024-03-04T16:33:51.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5zYN7urzp8gpP40yG4QYRYt9n2k=', 'Content-Length': '0'} 2024-03-04T16:33:51.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'hfXmZko7NeFEk5eOt6OzuIOcw0ti-Kt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000452a75ab387dd1f8-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.924 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.924 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.924 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.925 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.925 DEBUG:boto:Method: GET 2024-03-04T16:33:51.925 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.925 DEBUG:boto:Data: 2024-03-04T16:33:51.925 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.925 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.925 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.925 DEBUG:boto:Params: {} 2024-03-04T16:33:51.925 DEBUG:boto:Token: None 2024-03-04T16:33:51.925 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e24f2be3ee5bd487-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.929 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.929 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:51.929 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:51.929 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.930 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:51.930 DEBUG:boto:Method: GET 2024-03-04T16:33:51.930 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:51.930 DEBUG:boto:Data: 2024-03-04T16:33:51.930 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.930 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.930 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.930 DEBUG:boto:Params: {} 2024-03-04T16:33:51.930 DEBUG:boto:Token: None 2024-03-04T16:33:51.930 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/ 2024-03-04T16:33:51.931 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU= 2024-03-04T16:33:51.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:imLUVdot7iJwMZwiLF1xG4Yb5gU=', 'Content-Length': '0'} 2024-03-04T16:33:51.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93353b1acadc725-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.976 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:51.977 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.977 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.977 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.977 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.977 DEBUG:boto:Data: 2024-03-04T16:33:51.977 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.977 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.977 DEBUG:boto:Params: {} 2024-03-04T16:33:51.978 DEBUG:boto:Token: None 2024-03-04T16:33:51.978 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617066ecaf2a720d-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.982 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.982 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.982 DEBUG:boto:Method: HEAD 2024-03-04T16:33:51.982 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:51.982 DEBUG:boto:Data: 2024-03-04T16:33:51.982 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.982 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:51.982 DEBUG:boto:Port: 8000 2024-03-04T16:33:51.982 DEBUG:boto:Params: {} 2024-03-04T16:33:51.982 DEBUG:boto:Token: None 2024-03-04T16:33:51.983 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj 2024-03-04T16:33:51.983 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0= 2024-03-04T16:33:51.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MaBqmLQsDzYyM0J3qJaAEsDVxN0=', 'Content-Length': '0'} 2024-03-04T16:33:51.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e8097a31ec5e74b-0065e5f7ef-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:51.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:51.987 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:51.987 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:51.987 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.987 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.987 DEBUG:boto:Method: GET 2024-03-04T16:33:51.987 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.987 DEBUG:boto:Data: 2024-03-04T16:33:51.987 DEBUG:boto:Headers: {} 2024-03-04T16:33:51.987 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:51.987 DEBUG:boto:Port: 8001 2024-03-04T16:33:51.987 DEBUG:boto:Params: {} 2024-03-04T16:33:51.988 DEBUG:boto:Token: None 2024-03-04T16:33:51.988 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:51 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:51.988 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0= 2024-03-04T16:33:51.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LLRCFHdj7uctONkxLbzKq33ohn0=', 'Content-Length': '0'} 2024-03-04T16:33:51.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f094852f8b224f2-0065e5f7ef-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.036 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.036 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.036 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.037 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.037 DEBUG:boto:Method: GET 2024-03-04T16:33:52.037 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.037 DEBUG:boto:Data: 2024-03-04T16:33:52.037 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.037 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.037 DEBUG:boto:Params: {} 2024-03-04T16:33:52.037 DEBUG:boto:Token: None 2024-03-04T16:33:52.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fbb13eec7486877-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.040 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.040 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.040 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.040 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.041 DEBUG:boto:Method: GET 2024-03-04T16:33:52.041 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.041 DEBUG:boto:Data: 2024-03-04T16:33:52.041 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.041 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.041 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.041 DEBUG:boto:Params: {} 2024-03-04T16:33:52.041 DEBUG:boto:Token: None 2024-03-04T16:33:52.041 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.041 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f652aa678123b8a-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.044 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.045 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.045 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.045 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.045 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.045 DEBUG:boto:Method: GET 2024-03-04T16:33:52.045 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.046 DEBUG:boto:Data: 2024-03-04T16:33:52.046 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.046 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.046 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.046 DEBUG:boto:Params: {} 2024-03-04T16:33:52.046 DEBUG:boto:Token: None 2024-03-04T16:33:52.046 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eed25848ac0ab4a-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.049 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.050 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.050 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.050 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.050 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.050 DEBUG:boto:Method: GET 2024-03-04T16:33:52.050 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.050 DEBUG:boto:Data: 2024-03-04T16:33:52.050 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.050 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.050 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.050 DEBUG:boto:Params: {} 2024-03-04T16:33:52.051 DEBUG:boto:Token: None 2024-03-04T16:33:52.051 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.051 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w= 2024-03-04T16:33:52.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w=', 'Content-Length': '0'} 2024-03-04T16:33:52.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009f2e281aa1e09c9-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.103 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.103 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.103 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.103 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.103 DEBUG:boto:Method: GET 2024-03-04T16:33:52.103 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.103 DEBUG:boto:Data: 2024-03-04T16:33:52.104 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.104 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.104 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.104 DEBUG:boto:Params: {} 2024-03-04T16:33:52.104 DEBUG:boto:Token: None 2024-03-04T16:33:52.104 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.104 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w= 2024-03-04T16:33:52.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w=', 'Content-Length': '0'} 2024-03-04T16:33:52.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce2562da386784f4-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.148 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.148 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.148 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.149 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.149 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.149 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.150 DEBUG:boto:Data: 2024-03-04T16:33:52.150 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.150 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.150 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.150 DEBUG:boto:Params: {} 2024-03-04T16:33:52.151 DEBUG:boto:Token: None 2024-03-04T16:33:52.151 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.151 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/GrRsVcn50XusZRM9s8Eu/57ajk= 2024-03-04T16:33:52.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/GrRsVcn50XusZRM9s8Eu/57ajk=', 'Content-Length': '0'} 2024-03-04T16:33:52.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009108dd72f5e43c8d-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.154 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.154 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.155 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.155 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.155 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.155 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.155 DEBUG:boto:Data: 2024-03-04T16:33:52.156 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.156 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.156 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.156 DEBUG:boto:Params: {} 2024-03-04T16:33:52.157 DEBUG:boto:Token: None 2024-03-04T16:33:52.157 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.157 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/GrRsVcn50XusZRM9s8Eu/57ajk= 2024-03-04T16:33:52.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/GrRsVcn50XusZRM9s8Eu/57ajk=', 'Content-Length': '0'} 2024-03-04T16:33:52.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ca2eed1674efe8d-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.160 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.161 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.161 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.161 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.161 DEBUG:boto:Method: GET 2024-03-04T16:33:52.161 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.162 DEBUG:boto:Data: 2024-03-04T16:33:52.162 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.162 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.162 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.162 DEBUG:boto:Params: {} 2024-03-04T16:33:52.163 DEBUG:boto:Token: None 2024-03-04T16:33:52.163 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.163 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w= 2024-03-04T16:33:52.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w=', 'Content-Length': '0'} 2024-03-04T16:33:52.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224a79ec05eba8b0-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.213 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.213 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.213 DEBUG:boto:path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.213 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.213 DEBUG:boto:Method: GET 2024-03-04T16:33:52.213 DEBUG:boto:Path: /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.213 DEBUG:boto:Data: 2024-03-04T16:33:52.213 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.213 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.213 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.214 DEBUG:boto:Params: {} 2024-03-04T16:33:52.214 DEBUG:boto:Token: None 2024-03-04T16:33:52.214 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=objybm.zcpg4NCgTW0qlMotVwsu1Qap 2024-03-04T16:33:52.214 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w= 2024-03-04T16:33:52.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DMzT2FdQ+bzs6ggrOdFIrsv/N6w=', 'Content-Length': '0'} 2024-03-04T16:33:52.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'objybm.zcpg4NCgTW0qlMotVwsu1Qap'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073026264f70bde68-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.260 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.260 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.260 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.261 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.261 DEBUG:boto:Method: GET 2024-03-04T16:33:52.261 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.261 DEBUG:boto:Data: 2024-03-04T16:33:52.261 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.261 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.261 DEBUG:boto:Params: {} 2024-03-04T16:33:52.261 DEBUG:boto:Token: None 2024-03-04T16:33:52.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.262 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3258e6080b94e57-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.265 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.265 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.265 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.265 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.266 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.266 DEBUG:boto:Method: GET 2024-03-04T16:33:52.266 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.266 DEBUG:boto:Data: 2024-03-04T16:33:52.266 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.266 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.266 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.266 DEBUG:boto:Params: {} 2024-03-04T16:33:52.266 DEBUG:boto:Token: None 2024-03-04T16:33:52.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbd0b52117074e06-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.269 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.270 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.270 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.270 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.270 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.270 DEBUG:boto:Data: 2024-03-04T16:33:52.270 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.270 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.271 DEBUG:boto:Params: {} 2024-03-04T16:33:52.271 DEBUG:boto:Token: None 2024-03-04T16:33:52.271 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc4d275889730b21-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.274 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.274 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.275 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.275 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.275 DEBUG:boto:Data: 2024-03-04T16:33:52.275 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.275 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.275 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.275 DEBUG:boto:Params: {} 2024-03-04T16:33:52.275 DEBUG:boto:Token: None 2024-03-04T16:33:52.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007df71556d9597999-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.279 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.279 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.279 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.280 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.280 DEBUG:boto:Method: GET 2024-03-04T16:33:52.280 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.280 DEBUG:boto:Data: 2024-03-04T16:33:52.280 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.280 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.280 DEBUG:boto:Params: {} 2024-03-04T16:33:52.280 DEBUG:boto:Token: None 2024-03-04T16:33:52.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.281 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c48e8e3243fc0a8e-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.328 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.328 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.328 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.328 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.329 DEBUG:boto:Method: GET 2024-03-04T16:33:52.329 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.329 DEBUG:boto:Data: 2024-03-04T16:33:52.329 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.329 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.329 DEBUG:boto:Params: {} 2024-03-04T16:33:52.329 DEBUG:boto:Token: None 2024-03-04T16:33:52.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bac35243ef9787f-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.332 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.332 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.332 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.332 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.332 DEBUG:boto:Method: GET 2024-03-04T16:33:52.332 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.333 DEBUG:boto:Data: 2024-03-04T16:33:52.333 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.333 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.333 DEBUG:boto:Params: {} 2024-03-04T16:33:52.333 DEBUG:boto:Token: None 2024-03-04T16:33:52.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2eeda8492f61c5-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.336 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.337 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.337 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.337 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.337 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.337 DEBUG:boto:Method: GET 2024-03-04T16:33:52.337 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.337 DEBUG:boto:Data: 2024-03-04T16:33:52.337 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.337 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.337 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.337 DEBUG:boto:Params: {} 2024-03-04T16:33:52.338 DEBUG:boto:Token: None 2024-03-04T16:33:52.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca665d00e3495d0-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.341 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.342 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.342 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.342 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.342 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.342 DEBUG:boto:Method: GET 2024-03-04T16:33:52.342 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.342 DEBUG:boto:Data: 2024-03-04T16:33:52.342 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.342 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.342 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.343 DEBUG:boto:Params: {} 2024-03-04T16:33:52.343 DEBUG:boto:Token: None 2024-03-04T16:33:52.343 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.343 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ= 2024-03-04T16:33:52.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd58418f651337fb-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.345 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.345 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.345 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.345 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.345 DEBUG:boto:Method: GET 2024-03-04T16:33:52.345 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.345 DEBUG:boto:Data: 2024-03-04T16:33:52.346 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.346 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.346 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.346 DEBUG:boto:Params: {} 2024-03-04T16:33:52.346 DEBUG:boto:Token: None 2024-03-04T16:33:52.346 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.346 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ= 2024-03-04T16:33:52.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008666275d9fae8eed-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.397 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.397 DEBUG:boto:Data: 2024-03-04T16:33:52.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.397 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.397 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.397 DEBUG:boto:Params: {} 2024-03-04T16:33:52.397 DEBUG:boto:Token: None 2024-03-04T16:33:52.397 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ph8UWJ39LPGne+MeTIeHBWgq8mU= 2024-03-04T16:33:52.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ph8UWJ39LPGne+MeTIeHBWgq8mU=', 'Content-Length': '0'} 2024-03-04T16:33:52.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c7b48145187bf7a-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.400 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.400 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.400 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.400 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.400 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.400 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.400 DEBUG:boto:Data: 2024-03-04T16:33:52.400 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.401 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.401 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.401 DEBUG:boto:Params: {} 2024-03-04T16:33:52.401 DEBUG:boto:Token: None 2024-03-04T16:33:52.401 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ph8UWJ39LPGne+MeTIeHBWgq8mU= 2024-03-04T16:33:52.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ph8UWJ39LPGne+MeTIeHBWgq8mU=', 'Content-Length': '0'} 2024-03-04T16:33:52.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e5c6db752de6766-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.404 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.404 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.404 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.404 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.404 DEBUG:boto:Method: GET 2024-03-04T16:33:52.404 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.404 DEBUG:boto:Data: 2024-03-04T16:33:52.404 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.404 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.404 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.404 DEBUG:boto:Params: {} 2024-03-04T16:33:52.405 DEBUG:boto:Token: None 2024-03-04T16:33:52.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ= 2024-03-04T16:33:52.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003498b0e6d6aa1105-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.407 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.407 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.407 DEBUG:boto:path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.407 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.408 DEBUG:boto:Method: GET 2024-03-04T16:33:52.408 DEBUG:boto:Path: /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.408 DEBUG:boto:Data: 2024-03-04T16:33:52.408 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.408 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.408 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.408 DEBUG:boto:Params: {} 2024-03-04T16:33:52.408 DEBUG:boto:Token: None 2024-03-04T16:33:52.408 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=01uuPC1IwgjYFQiRu10-OMZvc7W7i9d 2024-03-04T16:33:52.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ= 2024-03-04T16:33:52.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:popDM9DEfP1niW37kXxnVnZfihQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '01uuPC1IwgjYFQiRu10-OMZvc7W7i9d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6f76f4021a0d01-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.452 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.452 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.453 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.453 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.453 DEBUG:boto:Method: GET 2024-03-04T16:33:52.453 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.453 DEBUG:boto:Data: 2024-03-04T16:33:52.453 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.453 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.453 DEBUG:boto:Params: {} 2024-03-04T16:33:52.453 DEBUG:boto:Token: None 2024-03-04T16:33:52.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004338fea598c997e3-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.457 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.457 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.457 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.457 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.458 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.458 DEBUG:boto:Method: GET 2024-03-04T16:33:52.458 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.458 DEBUG:boto:Data: 2024-03-04T16:33:52.458 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.458 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.458 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.458 DEBUG:boto:Params: {} 2024-03-04T16:33:52.458 DEBUG:boto:Token: None 2024-03-04T16:33:52.458 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a51d507dbe0911-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.462 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.462 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.462 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.462 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.462 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.462 DEBUG:boto:Data: 2024-03-04T16:33:52.462 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.462 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.463 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.463 DEBUG:boto:Params: {} 2024-03-04T16:33:52.463 DEBUG:boto:Token: None 2024-03-04T16:33:52.463 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.463 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081be5ada698621a3-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.467 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.467 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.467 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.467 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.467 DEBUG:boto:Data: 2024-03-04T16:33:52.467 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.467 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.467 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.467 DEBUG:boto:Params: {} 2024-03-04T16:33:52.468 DEBUG:boto:Token: None 2024-03-04T16:33:52.468 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005961bb7c593249c4-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.471 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.472 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.472 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.472 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.472 DEBUG:boto:Method: GET 2024-03-04T16:33:52.472 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.472 DEBUG:boto:Data: 2024-03-04T16:33:52.472 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.472 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.472 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.472 DEBUG:boto:Params: {} 2024-03-04T16:33:52.472 DEBUG:boto:Token: None 2024-03-04T16:33:52.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.473 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009642e5fe2b0a1cbd-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.516 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.516 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.516 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.517 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.517 DEBUG:boto:Method: GET 2024-03-04T16:33:52.517 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.517 DEBUG:boto:Data: 2024-03-04T16:33:52.517 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.517 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.517 DEBUG:boto:Params: {} 2024-03-04T16:33:52.517 DEBUG:boto:Token: None 2024-03-04T16:33:52.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066d1aa5282c5250e-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.520 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.520 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.520 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.520 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.520 DEBUG:boto:Method: GET 2024-03-04T16:33:52.521 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.521 DEBUG:boto:Data: 2024-03-04T16:33:52.521 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.521 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.521 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.521 DEBUG:boto:Params: {} 2024-03-04T16:33:52.521 DEBUG:boto:Token: None 2024-03-04T16:33:52.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b94b9f8343259f-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.524 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.525 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.525 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.525 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.525 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.525 DEBUG:boto:Method: GET 2024-03-04T16:33:52.525 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.525 DEBUG:boto:Data: 2024-03-04T16:33:52.525 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.525 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.526 DEBUG:boto:Params: {} 2024-03-04T16:33:52.526 DEBUG:boto:Token: None 2024-03-04T16:33:52.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d8b0c6c71c5c6a-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.529 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.529 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.529 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.530 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.530 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.530 DEBUG:boto:Method: GET 2024-03-04T16:33:52.530 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.530 DEBUG:boto:Data: 2024-03-04T16:33:52.530 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.530 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.530 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.530 DEBUG:boto:Params: {} 2024-03-04T16:33:52.530 DEBUG:boto:Token: None 2024-03-04T16:33:52.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo= 2024-03-04T16:33:52.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo=', 'Content-Length': '0'} 2024-03-04T16:33:52.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d37e4bebc35f1fa5-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.576 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.576 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.576 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.576 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.577 DEBUG:boto:Method: GET 2024-03-04T16:33:52.577 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.577 DEBUG:boto:Data: 2024-03-04T16:33:52.577 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.577 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.577 DEBUG:boto:Params: {} 2024-03-04T16:33:52.577 DEBUG:boto:Token: None 2024-03-04T16:33:52.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo= 2024-03-04T16:33:52.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo=', 'Content-Length': '0'} 2024-03-04T16:33:52.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1e756dbce5579dc-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.620 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.620 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.620 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.621 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.621 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.621 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.621 DEBUG:boto:Data: 2024-03-04T16:33:52.621 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.621 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.621 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.621 DEBUG:boto:Params: {} 2024-03-04T16:33:52.621 DEBUG:boto:Token: None 2024-03-04T16:33:52.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Juv/p6WyxvjZHGONzBGcc/shMHE= 2024-03-04T16:33:52.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Juv/p6WyxvjZHGONzBGcc/shMHE=', 'Content-Length': '0'} 2024-03-04T16:33:52.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034932f2103aa3bc3-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.624 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.624 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.624 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.624 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.624 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.624 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.624 DEBUG:boto:Data: 2024-03-04T16:33:52.624 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.624 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.624 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.625 DEBUG:boto:Params: {} 2024-03-04T16:33:52.625 DEBUG:boto:Token: None 2024-03-04T16:33:52.625 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Juv/p6WyxvjZHGONzBGcc/shMHE= 2024-03-04T16:33:52.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Juv/p6WyxvjZHGONzBGcc/shMHE=', 'Content-Length': '0'} 2024-03-04T16:33:52.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000377cd10b690e0c09-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.627 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.627 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.627 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.627 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.627 DEBUG:boto:Method: GET 2024-03-04T16:33:52.628 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.628 DEBUG:boto:Data: 2024-03-04T16:33:52.628 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.628 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.628 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.628 DEBUG:boto:Params: {} 2024-03-04T16:33:52.628 DEBUG:boto:Token: None 2024-03-04T16:33:52.628 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.628 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo= 2024-03-04T16:33:52.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo=', 'Content-Length': '0'} 2024-03-04T16:33:52.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5c9a781fd055d5b-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.672 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.673 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.673 DEBUG:boto:path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.673 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.673 DEBUG:boto:Method: GET 2024-03-04T16:33:52.673 DEBUG:boto:Path: /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.673 DEBUG:boto:Data: 2024-03-04T16:33:52.673 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.673 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.673 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.674 DEBUG:boto:Params: {} 2024-03-04T16:33:52.674 DEBUG:boto:Token: None 2024-03-04T16:33:52.674 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb 2024-03-04T16:33:52.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo= 2024-03-04T16:33:52.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qm7pZOQS4b2+qTRyo64ydwuwtKo=', 'Content-Length': '0'} 2024-03-04T16:33:52.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'i4ms9tGVGjdbOyxew4b4Yd0zBU8rzCb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053bd78842971bea5-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.720 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.720 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.721 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.721 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.721 DEBUG:boto:Method: GET 2024-03-04T16:33:52.721 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.721 DEBUG:boto:Data: 2024-03-04T16:33:52.721 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.721 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.721 DEBUG:boto:Params: {} 2024-03-04T16:33:52.721 DEBUG:boto:Token: None 2024-03-04T16:33:52.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d29f2a1654a2d57-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.725 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.725 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.726 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.726 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.726 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.726 DEBUG:boto:Method: GET 2024-03-04T16:33:52.726 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.726 DEBUG:boto:Data: 2024-03-04T16:33:52.726 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.726 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.726 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.726 DEBUG:boto:Params: {} 2024-03-04T16:33:52.726 DEBUG:boto:Token: None 2024-03-04T16:33:52.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b7eac05a4c1d1c-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.729 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.730 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.730 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.730 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.730 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.730 DEBUG:boto:Data: 2024-03-04T16:33:52.731 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.731 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.731 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.731 DEBUG:boto:Params: {} 2024-03-04T16:33:52.731 DEBUG:boto:Token: None 2024-03-04T16:33:52.731 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.731 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014225832e50fee0b-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.735 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.735 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.735 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:52.735 DEBUG:boto:Data: 2024-03-04T16:33:52.735 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.735 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.735 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.735 DEBUG:boto:Params: {} 2024-03-04T16:33:52.735 DEBUG:boto:Token: None 2024-03-04T16:33:52.736 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj 2024-03-04T16:33:52.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4= 2024-03-04T16:33:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KSx8ooabVhiOkdDrKuaSGtZinA4=', 'Content-Length': '0'} 2024-03-04T16:33:52.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcee3cbd05f5149d-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.739 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.739 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.739 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.740 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.740 DEBUG:boto:Method: GET 2024-03-04T16:33:52.740 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.740 DEBUG:boto:Data: 2024-03-04T16:33:52.740 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.740 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.740 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.740 DEBUG:boto:Params: {} 2024-03-04T16:33:52.740 DEBUG:boto:Token: None 2024-03-04T16:33:52.740 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.741 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000110c11f09aab9427-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.784 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.784 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.784 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.784 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.785 DEBUG:boto:Method: GET 2024-03-04T16:33:52.785 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.785 DEBUG:boto:Data: 2024-03-04T16:33:52.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.785 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.785 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.785 DEBUG:boto:Params: {} 2024-03-04T16:33:52.785 DEBUG:boto:Token: None 2024-03-04T16:33:52.786 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:52.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo= 2024-03-04T16:33:52.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvF45esm5UrfVjx9MqrSxFcDdNo=', 'Content-Length': '0'} 2024-03-04T16:33:52.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d91f8b1be766007-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.788 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.788 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.788 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.788 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.789 DEBUG:boto:Method: GET 2024-03-04T16:33:52.789 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.789 DEBUG:boto:Data: 2024-03-04T16:33:52.789 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.789 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.789 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.789 DEBUG:boto:Params: {} 2024-03-04T16:33:52.789 DEBUG:boto:Token: None 2024-03-04T16:33:52.789 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d7c9c01c9db8511-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.836 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.837 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.837 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.837 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.837 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.837 DEBUG:boto:Method: GET 2024-03-04T16:33:52.837 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.837 DEBUG:boto:Data: 2024-03-04T16:33:52.838 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.838 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.838 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.838 DEBUG:boto:Params: {} 2024-03-04T16:33:52.838 DEBUG:boto:Token: None 2024-03-04T16:33:52.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81780845d920fe2-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.841 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.841 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.842 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.842 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.842 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.842 DEBUG:boto:Method: GET 2024-03-04T16:33:52.842 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.842 DEBUG:boto:Data: 2024-03-04T16:33:52.842 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.842 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.842 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.842 DEBUG:boto:Params: {} 2024-03-04T16:33:52.842 DEBUG:boto:Token: None 2024-03-04T16:33:52.843 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.843 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U= 2024-03-04T16:33:52.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U=', 'Content-Length': '0'} 2024-03-04T16:33:52.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9477a3982377bf2-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.888 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.888 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.889 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.889 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.889 DEBUG:boto:Method: GET 2024-03-04T16:33:52.889 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.889 DEBUG:boto:Data: 2024-03-04T16:33:52.889 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.889 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.890 DEBUG:boto:Params: {} 2024-03-04T16:33:52.890 DEBUG:boto:Token: None 2024-03-04T16:33:52.890 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U= 2024-03-04T16:33:52.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U=', 'Content-Length': '0'} 2024-03-04T16:33:52.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6fa03bd5431d627-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.892 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.893 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.893 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.893 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.893 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.893 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.893 DEBUG:boto:Data: 2024-03-04T16:33:52.893 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.893 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.893 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.893 DEBUG:boto:Params: {} 2024-03-04T16:33:52.893 DEBUG:boto:Token: None 2024-03-04T16:33:52.894 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.894 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g3a2HYRYOgwO7G5UQKUDq6DCxDQ= 2024-03-04T16:33:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g3a2HYRYOgwO7G5UQKUDq6DCxDQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010da39548a6b22dc-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.896 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.896 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.896 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.896 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.896 DEBUG:boto:Method: HEAD 2024-03-04T16:33:52.896 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.896 DEBUG:boto:Data: 2024-03-04T16:33:52.897 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.897 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.897 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.897 DEBUG:boto:Params: {} 2024-03-04T16:33:52.897 DEBUG:boto:Token: None 2024-03-04T16:33:52.897 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.897 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:g3a2HYRYOgwO7G5UQKUDq6DCxDQ= 2024-03-04T16:33:52.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g3a2HYRYOgwO7G5UQKUDq6DCxDQ=', 'Content-Length': '0'} 2024-03-04T16:33:52.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4ddcb096705a605-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:52.900 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.900 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.900 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.900 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.900 DEBUG:boto:Method: GET 2024-03-04T16:33:52.900 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.900 DEBUG:boto:Data: 2024-03-04T16:33:52.900 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.900 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.900 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.901 DEBUG:boto:Params: {} 2024-03-04T16:33:52.901 DEBUG:boto:Token: None 2024-03-04T16:33:52.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.901 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U= 2024-03-04T16:33:52.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U=', 'Content-Length': '0'} 2024-03-04T16:33:52.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b53030f608fed026-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.944 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:52.944 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:52.944 DEBUG:boto:path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.944 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.945 DEBUG:boto:Method: GET 2024-03-04T16:33:52.945 DEBUG:boto:Path: /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.945 DEBUG:boto:Data: 2024-03-04T16:33:52.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.945 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.945 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.945 DEBUG:boto:Params: {} 2024-03-04T16:33:52.945 DEBUG:boto:Token: None 2024-03-04T16:33:52.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/obj?versionId=t8CpszqgdrbNp1tmFmKatQEXeCtSLVg 2024-03-04T16:33:52.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U= 2024-03-04T16:33:52.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fRyKKaWlO1TLKi9Nu8RDubTSC9U=', 'Content-Length': '0'} 2024-03-04T16:33:52.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 't8CpszqgdrbNp1tmFmKatQEXeCtSLVg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7f635ddaabffca7-0065e5f7f0-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.948 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.948 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.948 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.948 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.948 DEBUG:boto:Method: GET 2024-03-04T16:33:52.948 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.948 DEBUG:boto:Data: 2024-03-04T16:33:52.948 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.948 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:52.948 DEBUG:boto:Port: 8001 2024-03-04T16:33:52.949 DEBUG:boto:Params: {} 2024-03-04T16:33:52.949 DEBUG:boto:Token: None 2024-03-04T16:33:52.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:52.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aabe87e55aa0752b-0065e5f7f0-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:52.996 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:52.997 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:52.997 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:52.997 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.997 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:52.997 DEBUG:boto:Method: GET 2024-03-04T16:33:52.997 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:52.997 DEBUG:boto:Data: 2024-03-04T16:33:52.997 DEBUG:boto:Headers: {} 2024-03-04T16:33:52.997 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:52.997 DEBUG:boto:Port: 8000 2024-03-04T16:33:52.997 DEBUG:boto:Params: {} 2024-03-04T16:33:52.998 DEBUG:boto:Token: None 2024-03-04T16:33:52.998 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:52 GMT /sbnvju-22/ 2024-03-04T16:33:52.998 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q= 2024-03-04T16:33:52.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:52 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PRJlVQcKgf/sNMbE0JiiZ/VSW/Q=', 'Content-Length': '0'} 2024-03-04T16:33:53.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3e93d7e3be6fc55-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.001 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.001 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.001 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.002 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.002 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.002 DEBUG:boto:Data: 2024-03-04T16:33:53.002 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.002 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.002 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.002 DEBUG:boto:Params: {} 2024-03-04T16:33:53.002 DEBUG:boto:Token: None 2024-03-04T16:33:53.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.003 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034f91e0eca6516f8-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.007 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.007 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.007 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.007 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.007 DEBUG:boto:Data: 2024-03-04T16:33:53.007 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.007 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.008 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.008 DEBUG:boto:Params: {} 2024-03-04T16:33:53.008 DEBUG:boto:Token: None 2024-03-04T16:33:53.008 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.008 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c33a6e71559212f-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.012 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.012 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.012 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.012 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.012 DEBUG:boto:Method: GET 2024-03-04T16:33:53.012 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.012 DEBUG:boto:Data: 2024-03-04T16:33:53.012 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.012 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.012 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.012 DEBUG:boto:Params: {} 2024-03-04T16:33:53.013 DEBUG:boto:Token: None 2024-03-04T16:33:53.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.013 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c1a97f79b783e2-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.056 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.056 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.056 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.057 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.057 DEBUG:boto:Method: GET 2024-03-04T16:33:53.057 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.057 DEBUG:boto:Data: 2024-03-04T16:33:53.057 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.057 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.057 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.057 DEBUG:boto:Params: {} 2024-03-04T16:33:53.057 DEBUG:boto:Token: None 2024-03-04T16:33:53.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.058 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f3dadcef7c682b6-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.060 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.060 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.060 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.060 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.060 DEBUG:boto:Method: GET 2024-03-04T16:33:53.060 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.060 DEBUG:boto:Data: 2024-03-04T16:33:53.060 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.060 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.061 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.061 DEBUG:boto:Params: {} 2024-03-04T16:33:53.061 DEBUG:boto:Token: None 2024-03-04T16:33:53.061 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009aef1fe52673900-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.104 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.104 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.105 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.105 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.105 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.105 DEBUG:boto:Method: GET 2024-03-04T16:33:53.105 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.105 DEBUG:boto:Data: 2024-03-04T16:33:53.105 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.105 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.105 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.105 DEBUG:boto:Params: {} 2024-03-04T16:33:53.105 DEBUG:boto:Token: None 2024-03-04T16:33:53.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.107 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ae15808756b13e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.110 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.110 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.110 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.110 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.110 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.111 DEBUG:boto:Method: GET 2024-03-04T16:33:53.111 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.111 DEBUG:boto:Data: 2024-03-04T16:33:53.111 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.111 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.111 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.111 DEBUG:boto:Params: {} 2024-03-04T16:33:53.111 DEBUG:boto:Token: None 2024-03-04T16:33:53.111 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.111 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8= 2024-03-04T16:33:53.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8=', 'Content-Length': '0'} 2024-03-04T16:33:53.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eea39f86b5f1b74b-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.156 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.156 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.156 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.157 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.157 DEBUG:boto:Method: GET 2024-03-04T16:33:53.157 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.157 DEBUG:boto:Data: 2024-03-04T16:33:53.157 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.157 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.157 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.157 DEBUG:boto:Params: {} 2024-03-04T16:33:53.157 DEBUG:boto:Token: None 2024-03-04T16:33:53.157 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.158 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8= 2024-03-04T16:33:53.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8=', 'Content-Length': '0'} 2024-03-04T16:33:53.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5e911f71994024d-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.160 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.160 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.160 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.160 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.161 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.161 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.161 DEBUG:boto:Data: 2024-03-04T16:33:53.161 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.161 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.161 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.161 DEBUG:boto:Params: {} 2024-03-04T16:33:53.161 DEBUG:boto:Token: None 2024-03-04T16:33:53.161 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.161 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hq8O/lYqQw1sEmGvj9waSjjeHYc= 2024-03-04T16:33:53.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hq8O/lYqQw1sEmGvj9waSjjeHYc=', 'Content-Length': '0'} 2024-03-04T16:33:53.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000865a4505076bd7cc-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.164 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.164 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.164 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.164 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.164 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.164 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.164 DEBUG:boto:Data: 2024-03-04T16:33:53.164 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.164 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.165 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.165 DEBUG:boto:Params: {} 2024-03-04T16:33:53.165 DEBUG:boto:Token: None 2024-03-04T16:33:53.165 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.165 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hq8O/lYqQw1sEmGvj9waSjjeHYc= 2024-03-04T16:33:53.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hq8O/lYqQw1sEmGvj9waSjjeHYc=', 'Content-Length': '0'} 2024-03-04T16:33:53.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fa945df45aff33b-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.167 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.167 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.167 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.167 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.168 DEBUG:boto:Method: GET 2024-03-04T16:33:53.168 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.168 DEBUG:boto:Data: 2024-03-04T16:33:53.168 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.168 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.168 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.168 DEBUG:boto:Params: {} 2024-03-04T16:33:53.168 DEBUG:boto:Token: None 2024-03-04T16:33:53.168 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.168 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8= 2024-03-04T16:33:53.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8=', 'Content-Length': '0'} 2024-03-04T16:33:53.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008162a9cae36b0ae-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.212 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.212 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.212 DEBUG:boto:path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.212 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.213 DEBUG:boto:Method: GET 2024-03-04T16:33:53.213 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.213 DEBUG:boto:Data: 2024-03-04T16:33:53.213 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.213 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.213 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.213 DEBUG:boto:Params: {} 2024-03-04T16:33:53.213 DEBUG:boto:Token: None 2024-03-04T16:33:53.213 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=XUAluvGVKdo-73mqoSBEEUwxIHvrmM4 2024-03-04T16:33:53.213 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8= 2024-03-04T16:33:53.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f1iz5QBApA2ctS1rNGH545PVQT8=', 'Content-Length': '0'} 2024-03-04T16:33:53.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XUAluvGVKdo-73mqoSBEEUwxIHvrmM4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000633cf7b6e263ac49-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.216 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.216 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.216 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.216 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.218 DEBUG:boto:Method: GET 2024-03-04T16:33:53.218 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.218 DEBUG:boto:Data: 2024-03-04T16:33:53.218 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.219 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.219 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.219 DEBUG:boto:Params: {} 2024-03-04T16:33:53.219 DEBUG:boto:Token: None 2024-03-04T16:33:53.219 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.219 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb791a25099b569-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.264 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.265 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.265 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.265 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.265 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.265 DEBUG:boto:Method: GET 2024-03-04T16:33:53.265 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.265 DEBUG:boto:Data: 2024-03-04T16:33:53.265 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.265 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.265 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.266 DEBUG:boto:Params: {} 2024-03-04T16:33:53.266 DEBUG:boto:Token: None 2024-03-04T16:33:53.266 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.266 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c2404be90e825e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.269 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.269 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.269 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.269 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.269 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.269 DEBUG:boto:Data: 2024-03-04T16:33:53.270 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.270 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.270 DEBUG:boto:Params: {} 2024-03-04T16:33:53.270 DEBUG:boto:Token: None 2024-03-04T16:33:53.270 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035f257301aab7e81-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.274 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.274 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.274 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.274 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.274 DEBUG:boto:Data: 2024-03-04T16:33:53.275 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.275 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.275 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.275 DEBUG:boto:Params: {} 2024-03-04T16:33:53.275 DEBUG:boto:Token: None 2024-03-04T16:33:53.275 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f5883ddfb1272a0-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.279 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.279 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.279 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.279 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.279 DEBUG:boto:Method: GET 2024-03-04T16:33:53.279 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.279 DEBUG:boto:Data: 2024-03-04T16:33:53.280 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.280 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.280 DEBUG:boto:Params: {} 2024-03-04T16:33:53.280 DEBUG:boto:Token: None 2024-03-04T16:33:53.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092629054c05b9944-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.328 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.328 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.328 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.329 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.329 DEBUG:boto:Method: GET 2024-03-04T16:33:53.329 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.329 DEBUG:boto:Data: 2024-03-04T16:33:53.329 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.329 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.329 DEBUG:boto:Params: {} 2024-03-04T16:33:53.329 DEBUG:boto:Token: None 2024-03-04T16:33:53.329 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b788d77293d0a80b-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.332 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.332 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.332 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.332 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.333 DEBUG:boto:Method: GET 2024-03-04T16:33:53.333 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.333 DEBUG:boto:Data: 2024-03-04T16:33:53.333 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.333 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.333 DEBUG:boto:Params: {} 2024-03-04T16:33:53.333 DEBUG:boto:Token: None 2024-03-04T16:33:53.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.335 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007db5dcc86690c52-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.380 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.381 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.381 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.381 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.381 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.381 DEBUG:boto:Method: GET 2024-03-04T16:33:53.381 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.381 DEBUG:boto:Data: 2024-03-04T16:33:53.381 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.381 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.382 DEBUG:boto:Params: {} 2024-03-04T16:33:53.382 DEBUG:boto:Token: None 2024-03-04T16:33:53.382 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b80ff0e75ce838d-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.385 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.385 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.386 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.386 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.386 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.386 DEBUG:boto:Method: GET 2024-03-04T16:33:53.386 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.386 DEBUG:boto:Data: 2024-03-04T16:33:53.386 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.386 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.386 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.386 DEBUG:boto:Params: {} 2024-03-04T16:33:53.386 DEBUG:boto:Token: None 2024-03-04T16:33:53.387 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.387 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY= 2024-03-04T16:33:53.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY=', 'Content-Length': '0'} 2024-03-04T16:33:53.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e88858757cb1731e-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.389 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.389 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.389 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.389 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.389 DEBUG:boto:Method: GET 2024-03-04T16:33:53.389 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.389 DEBUG:boto:Data: 2024-03-04T16:33:53.390 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.390 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.390 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.390 DEBUG:boto:Params: {} 2024-03-04T16:33:53.390 DEBUG:boto:Token: None 2024-03-04T16:33:53.390 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.390 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY= 2024-03-04T16:33:53.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY=', 'Content-Length': '0'} 2024-03-04T16:33:53.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025be2311b4e8c191-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.393 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.393 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.393 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.393 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.393 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.393 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.393 DEBUG:boto:Data: 2024-03-04T16:33:53.393 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.393 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.393 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.393 DEBUG:boto:Params: {} 2024-03-04T16:33:53.394 DEBUG:boto:Token: None 2024-03-04T16:33:53.394 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.394 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vn/G/jEObNDdZ5IFRHDYCQ+AI2M= 2024-03-04T16:33:53.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vn/G/jEObNDdZ5IFRHDYCQ+AI2M=', 'Content-Length': '0'} 2024-03-04T16:33:53.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c44393ac501d0d85-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.396 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.396 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.396 DEBUG:boto:Data: 2024-03-04T16:33:53.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.397 DEBUG:boto:Params: {} 2024-03-04T16:33:53.397 DEBUG:boto:Token: None 2024-03-04T16:33:53.397 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.397 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vn/G/jEObNDdZ5IFRHDYCQ+AI2M= 2024-03-04T16:33:53.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vn/G/jEObNDdZ5IFRHDYCQ+AI2M=', 'Content-Length': '0'} 2024-03-04T16:33:53.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098a4d8c37a951b20-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.400 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.400 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.400 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.400 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.400 DEBUG:boto:Method: GET 2024-03-04T16:33:53.400 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.400 DEBUG:boto:Data: 2024-03-04T16:33:53.400 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.400 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.400 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.400 DEBUG:boto:Params: {} 2024-03-04T16:33:53.400 DEBUG:boto:Token: None 2024-03-04T16:33:53.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY= 2024-03-04T16:33:53.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY=', 'Content-Length': '0'} 2024-03-04T16:33:53.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b52be436547aa247-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.403 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.403 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.403 DEBUG:boto:path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.403 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.403 DEBUG:boto:Method: GET 2024-03-04T16:33:53.403 DEBUG:boto:Path: /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.403 DEBUG:boto:Data: 2024-03-04T16:33:53.404 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.404 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.404 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.404 DEBUG:boto:Params: {} 2024-03-04T16:33:53.404 DEBUG:boto:Token: None 2024-03-04T16:33:53.404 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt 2024-03-04T16:33:53.404 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY= 2024-03-04T16:33:53.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+Ro1M4TJjOPPV0uguQbUaQQO/fY=', 'Content-Length': '0'} 2024-03-04T16:33:53.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'a2H67AHZ7-ZbPRGTiQGJV-y.JdDILOt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6289c036f96b224-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.448 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.448 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.448 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.448 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.449 DEBUG:boto:Method: GET 2024-03-04T16:33:53.449 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.449 DEBUG:boto:Data: 2024-03-04T16:33:53.449 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.449 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.449 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.449 DEBUG:boto:Params: {} 2024-03-04T16:33:53.449 DEBUG:boto:Token: None 2024-03-04T16:33:53.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a16f91bcc1eb919-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.453 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.453 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.453 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.453 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.453 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.453 DEBUG:boto:Method: GET 2024-03-04T16:33:53.453 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.453 DEBUG:boto:Data: 2024-03-04T16:33:53.454 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.454 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.454 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.454 DEBUG:boto:Params: {} 2024-03-04T16:33:53.454 DEBUG:boto:Token: None 2024-03-04T16:33:53.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b9bc01b0e6c02f-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.457 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.457 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.458 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.458 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.458 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.458 DEBUG:boto:Data: 2024-03-04T16:33:53.458 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.458 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.458 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.458 DEBUG:boto:Params: {} 2024-03-04T16:33:53.458 DEBUG:boto:Token: None 2024-03-04T16:33:53.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f9aec92a9d86008-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.462 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.462 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.462 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.463 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.463 DEBUG:boto:Data: 2024-03-04T16:33:53.463 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.463 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.463 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.463 DEBUG:boto:Params: {} 2024-03-04T16:33:53.463 DEBUG:boto:Token: None 2024-03-04T16:33:53.463 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.463 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4f85d68279aa300-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.467 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.467 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.467 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.467 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.467 DEBUG:boto:Method: GET 2024-03-04T16:33:53.467 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.467 DEBUG:boto:Data: 2024-03-04T16:33:53.468 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.468 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.468 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.468 DEBUG:boto:Params: {} 2024-03-04T16:33:53.468 DEBUG:boto:Token: None 2024-03-04T16:33:53.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000604cf5c0b802362a-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.512 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.512 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.512 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.513 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.513 DEBUG:boto:Method: GET 2024-03-04T16:33:53.513 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.513 DEBUG:boto:Data: 2024-03-04T16:33:53.513 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.513 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.513 DEBUG:boto:Params: {} 2024-03-04T16:33:53.513 DEBUG:boto:Token: None 2024-03-04T16:33:53.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da4973562f440ae6-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.516 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.516 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.516 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.516 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.516 DEBUG:boto:Method: GET 2024-03-04T16:33:53.516 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.517 DEBUG:boto:Data: 2024-03-04T16:33:53.517 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.517 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.517 DEBUG:boto:Params: {} 2024-03-04T16:33:53.517 DEBUG:boto:Token: None 2024-03-04T16:33:53.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.517 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003300ce4027c7161c-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.520 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.521 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.521 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.521 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.521 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.521 DEBUG:boto:Method: GET 2024-03-04T16:33:53.521 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.521 DEBUG:boto:Data: 2024-03-04T16:33:53.521 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.521 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.521 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.522 DEBUG:boto:Params: {} 2024-03-04T16:33:53.522 DEBUG:boto:Token: None 2024-03-04T16:33:53.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.522 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe8eab3c96e4600-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.525 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.525 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.525 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.525 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.526 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.526 DEBUG:boto:Method: GET 2024-03-04T16:33:53.526 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.526 DEBUG:boto:Data: 2024-03-04T16:33:53.526 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.526 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.526 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.526 DEBUG:boto:Params: {} 2024-03-04T16:33:53.526 DEBUG:boto:Token: None 2024-03-04T16:33:53.527 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo= 2024-03-04T16:33:53.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo=', 'Content-Length': '0'} 2024-03-04T16:33:53.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014b61cba2d58144e-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.572 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.572 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.572 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.572 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.573 DEBUG:boto:Method: GET 2024-03-04T16:33:53.573 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.573 DEBUG:boto:Data: 2024-03-04T16:33:53.573 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.573 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.573 DEBUG:boto:Params: {} 2024-03-04T16:33:53.573 DEBUG:boto:Token: None 2024-03-04T16:33:53.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo= 2024-03-04T16:33:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo=', 'Content-Length': '0'} 2024-03-04T16:33:53.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfce3dbe9b39ba35-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.616 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.616 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.616 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.616 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.617 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.617 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.617 DEBUG:boto:Data: 2024-03-04T16:33:53.617 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.617 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.617 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.617 DEBUG:boto:Params: {} 2024-03-04T16:33:53.617 DEBUG:boto:Token: None 2024-03-04T16:33:53.617 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aHeDuzPQuQmym+hsP+cLGP1FmnI= 2024-03-04T16:33:53.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHeDuzPQuQmym+hsP+cLGP1FmnI=', 'Content-Length': '0'} 2024-03-04T16:33:53.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb3045352f61e3b2-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.620 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.620 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.620 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.620 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.620 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.620 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.620 DEBUG:boto:Data: 2024-03-04T16:33:53.620 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.621 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.621 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.621 DEBUG:boto:Params: {} 2024-03-04T16:33:53.621 DEBUG:boto:Token: None 2024-03-04T16:33:53.621 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.621 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aHeDuzPQuQmym+hsP+cLGP1FmnI= 2024-03-04T16:33:53.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHeDuzPQuQmym+hsP+cLGP1FmnI=', 'Content-Length': '0'} 2024-03-04T16:33:53.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058f5ae96ef5d1e7e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.623 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.623 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.623 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.623 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.624 DEBUG:boto:Method: GET 2024-03-04T16:33:53.624 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.624 DEBUG:boto:Data: 2024-03-04T16:33:53.624 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.624 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.624 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.624 DEBUG:boto:Params: {} 2024-03-04T16:33:53.624 DEBUG:boto:Token: None 2024-03-04T16:33:53.624 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.624 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo= 2024-03-04T16:33:53.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo=', 'Content-Length': '0'} 2024-03-04T16:33:53.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046c82ac6e3c14013-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.668 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.668 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.668 DEBUG:boto:path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.668 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.669 DEBUG:boto:Method: GET 2024-03-04T16:33:53.669 DEBUG:boto:Path: /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.669 DEBUG:boto:Data: 2024-03-04T16:33:53.669 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.669 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.669 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.669 DEBUG:boto:Params: {} 2024-03-04T16:33:53.669 DEBUG:boto:Token: None 2024-03-04T16:33:53.669 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=gNT65zfABYfwW89PXHXbF.jE4EXc0Wl 2024-03-04T16:33:53.670 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo= 2024-03-04T16:33:53.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:E2kqOJobF2FWGxx+OSnj6s66KTo=', 'Content-Length': '0'} 2024-03-04T16:33:53.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'gNT65zfABYfwW89PXHXbF.jE4EXc0Wl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e451d007dd4011e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.716 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.716 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.716 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.716 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.717 DEBUG:boto:Method: GET 2024-03-04T16:33:53.717 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.717 DEBUG:boto:Data: 2024-03-04T16:33:53.717 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.717 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.717 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.717 DEBUG:boto:Params: {} 2024-03-04T16:33:53.717 DEBUG:boto:Token: None 2024-03-04T16:33:53.717 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.717 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20eb7f24065ab8d-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.720 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.721 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.721 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.721 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.721 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.721 DEBUG:boto:Method: GET 2024-03-04T16:33:53.721 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.721 DEBUG:boto:Data: 2024-03-04T16:33:53.722 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.722 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.722 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.722 DEBUG:boto:Params: {} 2024-03-04T16:33:53.722 DEBUG:boto:Token: None 2024-03-04T16:33:53.722 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.722 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255e7c36a6e074e1-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.725 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.725 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.725 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.725 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.726 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.726 DEBUG:boto:Data: 2024-03-04T16:33:53.726 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.726 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.726 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.726 DEBUG:boto:Params: {} 2024-03-04T16:33:53.726 DEBUG:boto:Token: None 2024-03-04T16:33:53.726 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061fb85d0082fcadf-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.730 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.730 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.730 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.730 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.731 DEBUG:boto:Data: 2024-03-04T16:33:53.731 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.731 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.731 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.731 DEBUG:boto:Params: {} 2024-03-04T16:33:53.731 DEBUG:boto:Token: None 2024-03-04T16:33:53.731 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.731 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4237bc8ba9c6911-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.735 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.735 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.735 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.736 DEBUG:boto:Method: GET 2024-03-04T16:33:53.736 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.736 DEBUG:boto:Data: 2024-03-04T16:33:53.736 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.736 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.736 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.736 DEBUG:boto:Params: {} 2024-03-04T16:33:53.736 DEBUG:boto:Token: None 2024-03-04T16:33:53.736 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e0ed5e047e7ad1-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.780 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.780 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.781 DEBUG:boto:Method: GET 2024-03-04T16:33:53.781 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.781 DEBUG:boto:Data: 2024-03-04T16:33:53.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.781 DEBUG:boto:Params: {} 2024-03-04T16:33:53.781 DEBUG:boto:Token: None 2024-03-04T16:33:53.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.781 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d892596d7f7a0c-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.784 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.784 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.784 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.784 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.784 DEBUG:boto:Method: GET 2024-03-04T16:33:53.785 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.785 DEBUG:boto:Data: 2024-03-04T16:33:53.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.785 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.785 DEBUG:boto:Params: {} 2024-03-04T16:33:53.785 DEBUG:boto:Token: None 2024-03-04T16:33:53.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd06ec75cf06af6-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.832 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.833 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.833 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.833 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.833 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.833 DEBUG:boto:Method: GET 2024-03-04T16:33:53.833 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.833 DEBUG:boto:Data: 2024-03-04T16:33:53.833 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.833 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.833 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.834 DEBUG:boto:Params: {} 2024-03-04T16:33:53.834 DEBUG:boto:Token: None 2024-03-04T16:33:53.834 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.834 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef87a747a7f615e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.837 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.837 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.837 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.837 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.837 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.837 DEBUG:boto:Method: GET 2024-03-04T16:33:53.838 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.838 DEBUG:boto:Data: 2024-03-04T16:33:53.838 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.838 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.838 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.838 DEBUG:boto:Params: {} 2024-03-04T16:33:53.838 DEBUG:boto:Token: None 2024-03-04T16:33:53.838 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.838 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw= 2024-03-04T16:33:53.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw=', 'Content-Length': '0'} 2024-03-04T16:33:53.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078553df7675b53b7-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.841 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.841 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.841 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.841 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.841 DEBUG:boto:Method: GET 2024-03-04T16:33:53.841 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.841 DEBUG:boto:Data: 2024-03-04T16:33:53.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.842 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.842 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.842 DEBUG:boto:Params: {} 2024-03-04T16:33:53.842 DEBUG:boto:Token: None 2024-03-04T16:33:53.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw= 2024-03-04T16:33:53.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw=', 'Content-Length': '0'} 2024-03-04T16:33:53.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ceacd9a3b158f83e-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.888 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.888 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.888 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.888 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.889 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.889 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.889 DEBUG:boto:Data: 2024-03-04T16:33:53.889 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.889 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.889 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.889 DEBUG:boto:Params: {} 2024-03-04T16:33:53.889 DEBUG:boto:Token: None 2024-03-04T16:33:53.889 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.890 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m411QTi8WyhRkGarSzuf1/lr2xU= 2024-03-04T16:33:53.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m411QTi8WyhRkGarSzuf1/lr2xU=', 'Content-Length': '0'} 2024-03-04T16:33:53.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015e86c8144b5fa0e-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.892 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.892 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.892 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.892 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.892 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.892 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.892 DEBUG:boto:Data: 2024-03-04T16:33:53.893 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.893 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.893 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.893 DEBUG:boto:Params: {} 2024-03-04T16:33:53.893 DEBUG:boto:Token: None 2024-03-04T16:33:53.893 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.893 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m411QTi8WyhRkGarSzuf1/lr2xU= 2024-03-04T16:33:53.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m411QTi8WyhRkGarSzuf1/lr2xU=', 'Content-Length': '0'} 2024-03-04T16:33:53.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a5f8f806610961f-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.895 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.896 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.896 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.896 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.896 DEBUG:boto:Method: GET 2024-03-04T16:33:53.896 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.896 DEBUG:boto:Data: 2024-03-04T16:33:53.896 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.896 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.896 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.896 DEBUG:boto:Params: {} 2024-03-04T16:33:53.896 DEBUG:boto:Token: None 2024-03-04T16:33:53.897 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.897 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw= 2024-03-04T16:33:53.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw=', 'Content-Length': '0'} 2024-03-04T16:33:53.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d69d8d8528f9a5b8-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.899 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.899 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.899 DEBUG:boto:path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.899 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.899 DEBUG:boto:Method: GET 2024-03-04T16:33:53.899 DEBUG:boto:Path: /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.899 DEBUG:boto:Data: 2024-03-04T16:33:53.899 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.899 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.900 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.900 DEBUG:boto:Params: {} 2024-03-04T16:33:53.900 DEBUG:boto:Token: None 2024-03-04T16:33:53.900 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=04Bk218QitWUyZShNL.038YuwPvLhql 2024-03-04T16:33:53.900 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw= 2024-03-04T16:33:53.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Be8MOD+lpXLOPcX2fcT9evjartw=', 'Content-Length': '0'} 2024-03-04T16:33:53.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '04Bk218QitWUyZShNL.038YuwPvLhql'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e325b04bf23a822-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.944 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.944 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.944 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.944 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.945 DEBUG:boto:Method: GET 2024-03-04T16:33:53.945 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.945 DEBUG:boto:Data: 2024-03-04T16:33:53.945 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.945 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.945 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.945 DEBUG:boto:Params: {} 2024-03-04T16:33:53.945 DEBUG:boto:Token: None 2024-03-04T16:33:53.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6530ace7294ade-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.949 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.949 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:53.952 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:53.953 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.953 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:53.953 DEBUG:boto:Method: GET 2024-03-04T16:33:53.953 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:53.953 DEBUG:boto:Data: 2024-03-04T16:33:53.953 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.953 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.953 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.953 DEBUG:boto:Params: {} 2024-03-04T16:33:53.953 DEBUG:boto:Token: None 2024-03-04T16:33:53.954 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/ 2024-03-04T16:33:53.954 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI= 2024-03-04T16:33:53.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jKJKsUbqtaKUuYt47QM/bxhoxyI=', 'Content-Length': '0'} 2024-03-04T16:33:53.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005596991ccde377c5-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.957 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:53.957 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.958 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.958 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.958 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.958 DEBUG:boto:Data: 2024-03-04T16:33:53.958 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.958 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.958 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.958 DEBUG:boto:Params: {} 2024-03-04T16:33:53.958 DEBUG:boto:Token: None 2024-03-04T16:33:53.958 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.959 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cef1dd1a9611ddeb-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.962 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.962 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.962 DEBUG:boto:Method: HEAD 2024-03-04T16:33:53.962 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:53.963 DEBUG:boto:Data: 2024-03-04T16:33:53.963 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:53.963 DEBUG:boto:Port: 8000 2024-03-04T16:33:53.963 DEBUG:boto:Params: {} 2024-03-04T16:33:53.963 DEBUG:boto:Token: None 2024-03-04T16:33:53.963 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj 2024-03-04T16:33:53.963 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY= 2024-03-04T16:33:53.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0aAdwHmJCdr8orhVIUOjjssPUmY=', 'Content-Length': '0'} 2024-03-04T16:33:53.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030d5567848eb5db8-0065e5f7f1-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:53.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:53.967 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:53.967 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:53.967 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.967 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.967 DEBUG:boto:Method: GET 2024-03-04T16:33:53.967 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.967 DEBUG:boto:Data: 2024-03-04T16:33:53.967 DEBUG:boto:Headers: {} 2024-03-04T16:33:53.967 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:53.968 DEBUG:boto:Port: 8001 2024-03-04T16:33:53.968 DEBUG:boto:Params: {} 2024-03-04T16:33:53.968 DEBUG:boto:Token: None 2024-03-04T16:33:53.968 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:53 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:53.968 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw= 2024-03-04T16:33:53.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:53 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YLSV6lLNxqAp5bfRa+UHitr0Whw=', 'Content-Length': '0'} 2024-03-04T16:33:53.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005601a13e31fcf662-0065e5f7f1-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.012 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.012 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.012 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.012 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.013 DEBUG:boto:Method: GET 2024-03-04T16:33:54.013 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.013 DEBUG:boto:Data: 2024-03-04T16:33:54.013 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.013 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.013 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.013 DEBUG:boto:Params: {} 2024-03-04T16:33:54.013 DEBUG:boto:Token: None 2024-03-04T16:33:54.013 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000661a51b9fb6f7ebd-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.016 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.016 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.016 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.016 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.016 DEBUG:boto:Method: GET 2024-03-04T16:33:54.016 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.016 DEBUG:boto:Data: 2024-03-04T16:33:54.016 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.017 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.017 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.017 DEBUG:boto:Params: {} 2024-03-04T16:33:54.017 DEBUG:boto:Token: None 2024-03-04T16:33:54.017 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.017 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e79d743685ff3af7-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.020 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.020 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.021 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.021 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.021 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.021 DEBUG:boto:Method: GET 2024-03-04T16:33:54.021 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.021 DEBUG:boto:Data: 2024-03-04T16:33:54.021 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.021 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.021 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.021 DEBUG:boto:Params: {} 2024-03-04T16:33:54.021 DEBUG:boto:Token: None 2024-03-04T16:33:54.022 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed8a43b66acac3c-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.024 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.025 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.025 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.025 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.025 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.025 DEBUG:boto:Method: GET 2024-03-04T16:33:54.025 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.025 DEBUG:boto:Data: 2024-03-04T16:33:54.026 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.026 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.026 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.026 DEBUG:boto:Params: {} 2024-03-04T16:33:54.026 DEBUG:boto:Token: None 2024-03-04T16:33:54.026 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.026 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM= 2024-03-04T16:33:54.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM=', 'Content-Length': '0'} 2024-03-04T16:33:54.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006aa380a17e1788cf-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.076 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.076 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.076 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.076 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.077 DEBUG:boto:Method: GET 2024-03-04T16:33:54.077 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.077 DEBUG:boto:Data: 2024-03-04T16:33:54.077 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.077 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.077 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.077 DEBUG:boto:Params: {} 2024-03-04T16:33:54.077 DEBUG:boto:Token: None 2024-03-04T16:33:54.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM= 2024-03-04T16:33:54.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM=', 'Content-Length': '0'} 2024-03-04T16:33:54.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c7e0e08dda06121-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.080 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.080 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.080 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.080 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.080 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.080 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.080 DEBUG:boto:Data: 2024-03-04T16:33:54.081 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.081 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.081 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.081 DEBUG:boto:Params: {} 2024-03-04T16:33:54.081 DEBUG:boto:Token: None 2024-03-04T16:33:54.081 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.081 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iDYwf+T334rpZsB7Y6wi92pqxKY= 2024-03-04T16:33:54.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDYwf+T334rpZsB7Y6wi92pqxKY=', 'Content-Length': '0'} 2024-03-04T16:33:54.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e3e3a08b5afd2af-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.083 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.083 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.083 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.083 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.083 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.084 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.084 DEBUG:boto:Data: 2024-03-04T16:33:54.084 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.084 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.084 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.084 DEBUG:boto:Params: {} 2024-03-04T16:33:54.084 DEBUG:boto:Token: None 2024-03-04T16:33:54.084 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.084 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iDYwf+T334rpZsB7Y6wi92pqxKY= 2024-03-04T16:33:54.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDYwf+T334rpZsB7Y6wi92pqxKY=', 'Content-Length': '0'} 2024-03-04T16:33:54.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059896481b026f604-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.087 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.087 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.087 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.087 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.087 DEBUG:boto:Method: GET 2024-03-04T16:33:54.088 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.088 DEBUG:boto:Data: 2024-03-04T16:33:54.088 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.088 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.088 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.088 DEBUG:boto:Params: {} 2024-03-04T16:33:54.088 DEBUG:boto:Token: None 2024-03-04T16:33:54.088 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.088 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM= 2024-03-04T16:33:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM=', 'Content-Length': '0'} 2024-03-04T16:33:54.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ac65f32200f6dbf-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.136 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.136 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.136 DEBUG:boto:path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.136 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.136 DEBUG:boto:Method: GET 2024-03-04T16:33:54.136 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.137 DEBUG:boto:Data: 2024-03-04T16:33:54.137 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.137 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.137 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.137 DEBUG:boto:Params: {} 2024-03-04T16:33:54.137 DEBUG:boto:Token: None 2024-03-04T16:33:54.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BLi4aTmZK0uBvmzaBcObjwKVJqxtERd 2024-03-04T16:33:54.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM= 2024-03-04T16:33:54.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1iC5Nbq1lPhgU4nkLyh+BGAUnVM=', 'Content-Length': '0'} 2024-03-04T16:33:54.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BLi4aTmZK0uBvmzaBcObjwKVJqxtERd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc1f3ed6fe359e3-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.140 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.140 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.140 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.140 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.140 DEBUG:boto:Method: GET 2024-03-04T16:33:54.140 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.140 DEBUG:boto:Data: 2024-03-04T16:33:54.140 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.141 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.141 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.141 DEBUG:boto:Params: {} 2024-03-04T16:33:54.141 DEBUG:boto:Token: None 2024-03-04T16:33:54.141 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fd9ba66bf7980c-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.144 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.144 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.144 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.144 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.144 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.144 DEBUG:boto:Method: GET 2024-03-04T16:33:54.145 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.145 DEBUG:boto:Data: 2024-03-04T16:33:54.145 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.145 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.145 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.145 DEBUG:boto:Params: {} 2024-03-04T16:33:54.145 DEBUG:boto:Token: None 2024-03-04T16:33:54.145 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078550ce44e306b39-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.148 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.148 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.148 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.149 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.149 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.149 DEBUG:boto:Data: 2024-03-04T16:33:54.149 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.149 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.149 DEBUG:boto:Params: {} 2024-03-04T16:33:54.149 DEBUG:boto:Token: None 2024-03-04T16:33:54.149 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033e934575732a056-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.153 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.153 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.153 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.153 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.153 DEBUG:boto:Data: 2024-03-04T16:33:54.153 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.153 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.154 DEBUG:boto:Params: {} 2024-03-04T16:33:54.155 DEBUG:boto:Token: None 2024-03-04T16:33:54.155 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.155 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000049610121d9cf3b-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.159 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.159 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.159 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.159 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.159 DEBUG:boto:Method: GET 2024-03-04T16:33:54.159 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.159 DEBUG:boto:Data: 2024-03-04T16:33:54.159 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.160 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.160 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.160 DEBUG:boto:Params: {} 2024-03-04T16:33:54.160 DEBUG:boto:Token: None 2024-03-04T16:33:54.160 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.160 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fa0f3b6213e4f5a-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.204 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.204 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.204 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.204 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.204 DEBUG:boto:Method: GET 2024-03-04T16:33:54.204 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.204 DEBUG:boto:Data: 2024-03-04T16:33:54.205 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.205 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.205 DEBUG:boto:Params: {} 2024-03-04T16:33:54.205 DEBUG:boto:Token: None 2024-03-04T16:33:54.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eea77a04afdd59bb-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.207 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.207 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.208 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.208 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.208 DEBUG:boto:Method: GET 2024-03-04T16:33:54.208 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.208 DEBUG:boto:Data: 2024-03-04T16:33:54.208 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.208 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.208 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.208 DEBUG:boto:Params: {} 2024-03-04T16:33:54.208 DEBUG:boto:Token: None 2024-03-04T16:33:54.208 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.209 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3cef02cfebbcd58-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.252 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.252 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.252 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.253 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.253 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.253 DEBUG:boto:Method: GET 2024-03-04T16:33:54.253 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.253 DEBUG:boto:Data: 2024-03-04T16:33:54.253 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.253 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.253 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.253 DEBUG:boto:Params: {} 2024-03-04T16:33:54.253 DEBUG:boto:Token: None 2024-03-04T16:33:54.254 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ac6047db1a6aae-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.257 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.257 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.257 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.257 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.257 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.257 DEBUG:boto:Method: GET 2024-03-04T16:33:54.257 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.257 DEBUG:boto:Data: 2024-03-04T16:33:54.257 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.258 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.258 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.258 DEBUG:boto:Params: {} 2024-03-04T16:33:54.258 DEBUG:boto:Token: None 2024-03-04T16:33:54.258 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g= 2024-03-04T16:33:54.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g=', 'Content-Length': '0'} 2024-03-04T16:33:54.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006621207d79354dc7-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.308 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.308 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.308 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.308 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.309 DEBUG:boto:Method: GET 2024-03-04T16:33:54.309 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.309 DEBUG:boto:Data: 2024-03-04T16:33:54.309 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.309 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.309 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.309 DEBUG:boto:Params: {} 2024-03-04T16:33:54.309 DEBUG:boto:Token: None 2024-03-04T16:33:54.309 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g= 2024-03-04T16:33:54.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g=', 'Content-Length': '0'} 2024-03-04T16:33:54.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5cd1c5882cf4958-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.352 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.352 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.352 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.352 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.353 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.353 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.353 DEBUG:boto:Data: 2024-03-04T16:33:54.353 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.353 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.353 DEBUG:boto:Params: {} 2024-03-04T16:33:54.353 DEBUG:boto:Token: None 2024-03-04T16:33:54.353 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:janQPrwgI0AD9KWYGJbxmphxdB8= 2024-03-04T16:33:54.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:janQPrwgI0AD9KWYGJbxmphxdB8=', 'Content-Length': '0'} 2024-03-04T16:33:54.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e12517eb8ffa25a6-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.356 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.356 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.356 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.356 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.356 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.356 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.356 DEBUG:boto:Data: 2024-03-04T16:33:54.356 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.356 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.357 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.357 DEBUG:boto:Params: {} 2024-03-04T16:33:54.357 DEBUG:boto:Token: None 2024-03-04T16:33:54.357 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:janQPrwgI0AD9KWYGJbxmphxdB8= 2024-03-04T16:33:54.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:janQPrwgI0AD9KWYGJbxmphxdB8=', 'Content-Length': '0'} 2024-03-04T16:33:54.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ace581166b68928a-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.361 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.361 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.361 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.361 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.361 DEBUG:boto:Method: GET 2024-03-04T16:33:54.361 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.361 DEBUG:boto:Data: 2024-03-04T16:33:54.361 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.362 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.362 DEBUG:boto:Params: {} 2024-03-04T16:33:54.362 DEBUG:boto:Token: None 2024-03-04T16:33:54.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g= 2024-03-04T16:33:54.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g=', 'Content-Length': '0'} 2024-03-04T16:33:54.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9e717aafded2ae6-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.412 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.412 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.412 DEBUG:boto:path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.412 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.412 DEBUG:boto:Method: GET 2024-03-04T16:33:54.413 DEBUG:boto:Path: /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.413 DEBUG:boto:Data: 2024-03-04T16:33:54.413 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.413 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.413 DEBUG:boto:Params: {} 2024-03-04T16:33:54.413 DEBUG:boto:Token: None 2024-03-04T16:33:54.413 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=tWwOAgDLiLza3F6cTWddXwRcOE7WT1B 2024-03-04T16:33:54.413 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g= 2024-03-04T16:33:54.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SpUj54hnwtPwAF22M/3P1lSy40g=', 'Content-Length': '0'} 2024-03-04T16:33:54.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'tWwOAgDLiLza3F6cTWddXwRcOE7WT1B'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e58cf9c77eae294-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.456 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.456 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.456 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.457 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.457 DEBUG:boto:Method: GET 2024-03-04T16:33:54.457 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.457 DEBUG:boto:Data: 2024-03-04T16:33:54.457 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.457 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.457 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.457 DEBUG:boto:Params: {} 2024-03-04T16:33:54.457 DEBUG:boto:Token: None 2024-03-04T16:33:54.457 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01a45ee196466be-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.504 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.505 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.505 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.505 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.505 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.505 DEBUG:boto:Method: GET 2024-03-04T16:33:54.505 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.505 DEBUG:boto:Data: 2024-03-04T16:33:54.505 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.505 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.505 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.505 DEBUG:boto:Params: {} 2024-03-04T16:33:54.506 DEBUG:boto:Token: None 2024-03-04T16:33:54.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043f243289c666f7f-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.509 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.509 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.509 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.510 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.510 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.510 DEBUG:boto:Data: 2024-03-04T16:33:54.510 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.510 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.510 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.510 DEBUG:boto:Params: {} 2024-03-04T16:33:54.510 DEBUG:boto:Token: None 2024-03-04T16:33:54.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000234ba24ce7eab71c-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.514 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.514 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.515 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.515 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.515 DEBUG:boto:Data: 2024-03-04T16:33:54.515 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.515 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.515 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.515 DEBUG:boto:Params: {} 2024-03-04T16:33:54.515 DEBUG:boto:Token: None 2024-03-04T16:33:54.515 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3ad4d47fea7ea7c-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.519 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.519 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.519 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.519 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.519 DEBUG:boto:Method: GET 2024-03-04T16:33:54.520 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.520 DEBUG:boto:Data: 2024-03-04T16:33:54.520 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.520 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.520 DEBUG:boto:Params: {} 2024-03-04T16:33:54.520 DEBUG:boto:Token: None 2024-03-04T16:33:54.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e0f07896133eaba-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.568 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.568 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.568 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.568 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.569 DEBUG:boto:Method: GET 2024-03-04T16:33:54.569 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.569 DEBUG:boto:Data: 2024-03-04T16:33:54.569 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.569 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.569 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.569 DEBUG:boto:Params: {} 2024-03-04T16:33:54.569 DEBUG:boto:Token: None 2024-03-04T16:33:54.569 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.569 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2112d75433d084e-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.572 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.572 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.572 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.572 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.572 DEBUG:boto:Method: GET 2024-03-04T16:33:54.572 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.572 DEBUG:boto:Data: 2024-03-04T16:33:54.573 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.573 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.573 DEBUG:boto:Params: {} 2024-03-04T16:33:54.573 DEBUG:boto:Token: None 2024-03-04T16:33:54.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495f9d90a345cf20-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.624 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.625 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.625 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.625 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.625 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.625 DEBUG:boto:Method: GET 2024-03-04T16:33:54.626 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.626 DEBUG:boto:Data: 2024-03-04T16:33:54.626 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.626 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.626 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.626 DEBUG:boto:Params: {} 2024-03-04T16:33:54.627 DEBUG:boto:Token: None 2024-03-04T16:33:54.627 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.627 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b3bc7d4968b3729-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.630 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.631 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.631 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.631 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.631 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.632 DEBUG:boto:Method: GET 2024-03-04T16:33:54.632 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.632 DEBUG:boto:Data: 2024-03-04T16:33:54.632 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.632 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.633 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.633 DEBUG:boto:Params: {} 2024-03-04T16:33:54.633 DEBUG:boto:Token: None 2024-03-04T16:33:54.633 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I= 2024-03-04T16:33:54.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I=', 'Content-Length': '0'} 2024-03-04T16:33:54.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff998ce8f581ceaa-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.676 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.676 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.676 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.676 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.677 DEBUG:boto:Method: GET 2024-03-04T16:33:54.677 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.677 DEBUG:boto:Data: 2024-03-04T16:33:54.677 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.677 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.677 DEBUG:boto:Params: {} 2024-03-04T16:33:54.677 DEBUG:boto:Token: None 2024-03-04T16:33:54.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I= 2024-03-04T16:33:54.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I=', 'Content-Length': '0'} 2024-03-04T16:33:54.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000676b502acf0c9e1b-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.680 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.680 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.681 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.681 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.681 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.681 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.681 DEBUG:boto:Data: 2024-03-04T16:33:54.681 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.681 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.681 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.681 DEBUG:boto:Params: {} 2024-03-04T16:33:54.681 DEBUG:boto:Token: None 2024-03-04T16:33:54.682 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.682 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tWQprPfiIv/W2BYXTW2zlI3us8g= 2024-03-04T16:33:54.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tWQprPfiIv/W2BYXTW2zlI3us8g=', 'Content-Length': '0'} 2024-03-04T16:33:54.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c3f2c947e1445a4-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.684 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.684 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.684 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.684 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.684 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.684 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.684 DEBUG:boto:Data: 2024-03-04T16:33:54.685 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.685 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.685 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.685 DEBUG:boto:Params: {} 2024-03-04T16:33:54.685 DEBUG:boto:Token: None 2024-03-04T16:33:54.685 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.685 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tWQprPfiIv/W2BYXTW2zlI3us8g= 2024-03-04T16:33:54.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tWQprPfiIv/W2BYXTW2zlI3us8g=', 'Content-Length': '0'} 2024-03-04T16:33:54.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000998655061e595066-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.687 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.687 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.688 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.688 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.688 DEBUG:boto:Method: GET 2024-03-04T16:33:54.688 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.688 DEBUG:boto:Data: 2024-03-04T16:33:54.688 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.688 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.688 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.688 DEBUG:boto:Params: {} 2024-03-04T16:33:54.688 DEBUG:boto:Token: None 2024-03-04T16:33:54.688 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.689 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I= 2024-03-04T16:33:54.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I=', 'Content-Length': '0'} 2024-03-04T16:33:54.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007b8582a9067855a-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.732 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.732 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.732 DEBUG:boto:path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.733 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.733 DEBUG:boto:Method: GET 2024-03-04T16:33:54.733 DEBUG:boto:Path: /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.733 DEBUG:boto:Data: 2024-03-04T16:33:54.733 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.733 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.733 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.733 DEBUG:boto:Params: {} 2024-03-04T16:33:54.733 DEBUG:boto:Token: None 2024-03-04T16:33:54.733 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv 2024-03-04T16:33:54.734 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I= 2024-03-04T16:33:54.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7HjcYQIPFpLarOi21JkviA4TF6I=', 'Content-Length': '0'} 2024-03-04T16:33:54.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '.zVNmA1NmzhFLJMBFCMm1Uq27Zq9WGv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a75461318c2325a2-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.736 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.736 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.736 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.736 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.736 DEBUG:boto:Method: GET 2024-03-04T16:33:54.736 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.737 DEBUG:boto:Data: 2024-03-04T16:33:54.737 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.737 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.737 DEBUG:boto:Params: {} 2024-03-04T16:33:54.737 DEBUG:boto:Token: None 2024-03-04T16:33:54.737 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.738 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068b83ba10bc65332-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.788 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.789 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.789 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.789 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.789 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.789 DEBUG:boto:Method: GET 2024-03-04T16:33:54.789 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.789 DEBUG:boto:Data: 2024-03-04T16:33:54.789 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.789 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.789 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.789 DEBUG:boto:Params: {} 2024-03-04T16:33:54.790 DEBUG:boto:Token: None 2024-03-04T16:33:54.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a775484c6be8c492-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.793 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.793 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.793 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.794 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.794 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.794 DEBUG:boto:Data: 2024-03-04T16:33:54.794 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.794 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.794 DEBUG:boto:Params: {} 2024-03-04T16:33:54.794 DEBUG:boto:Token: None 2024-03-04T16:33:54.794 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.794 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000463038faf71a3825-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.798 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.798 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.798 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.798 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:54.798 DEBUG:boto:Data: 2024-03-04T16:33:54.799 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.799 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.799 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.799 DEBUG:boto:Params: {} 2024-03-04T16:33:54.799 DEBUG:boto:Token: None 2024-03-04T16:33:54.799 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj 2024-03-04T16:33:54.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8= 2024-03-04T16:33:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L7rqPTHwnB47aCa5nk3vRLqjsu8=', 'Content-Length': '0'} 2024-03-04T16:33:54.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c71225d7d72171-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.803 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.803 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.803 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.803 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.803 DEBUG:boto:Method: GET 2024-03-04T16:33:54.803 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.803 DEBUG:boto:Data: 2024-03-04T16:33:54.804 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.804 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.804 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.804 DEBUG:boto:Params: {} 2024-03-04T16:33:54.804 DEBUG:boto:Token: None 2024-03-04T16:33:54.804 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.805 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c6f37352167bb60-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.852 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.853 DEBUG:boto:Method: GET 2024-03-04T16:33:54.853 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.853 DEBUG:boto:Data: 2024-03-04T16:33:54.853 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.853 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.853 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.853 DEBUG:boto:Params: {} 2024-03-04T16:33:54.853 DEBUG:boto:Token: None 2024-03-04T16:33:54.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:54.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE= 2024-03-04T16:33:54.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iLyywoFKcgOyFI7iwvtD5EbrbiE=', 'Content-Length': '0'} 2024-03-04T16:33:54.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079c6196792451391-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.856 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.856 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.856 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.856 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.856 DEBUG:boto:Method: GET 2024-03-04T16:33:54.856 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.857 DEBUG:boto:Data: 2024-03-04T16:33:54.857 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.857 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.857 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.857 DEBUG:boto:Params: {} 2024-03-04T16:33:54.857 DEBUG:boto:Token: None 2024-03-04T16:33:54.857 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e422828b9ad07f3-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.900 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.901 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:54.901 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:54.902 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.902 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:54.902 DEBUG:boto:Method: GET 2024-03-04T16:33:54.902 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:54.903 DEBUG:boto:Data: 2024-03-04T16:33:54.903 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.903 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.903 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.903 DEBUG:boto:Params: {} 2024-03-04T16:33:54.904 DEBUG:boto:Token: None 2024-03-04T16:33:54.904 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/ 2024-03-04T16:33:54.905 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU= 2024-03-04T16:33:54.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DqVX5HQAQgA/LWR64V+JsDkJ3JU=', 'Content-Length': '0'} 2024-03-04T16:33:54.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001028021c24a54250-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.908 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:54.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.909 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.909 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.909 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.910 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.910 DEBUG:boto:Method: GET 2024-03-04T16:33:54.910 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.910 DEBUG:boto:Data: 2024-03-04T16:33:54.910 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.910 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.911 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.911 DEBUG:boto:Params: {} 2024-03-04T16:33:54.911 DEBUG:boto:Token: None 2024-03-04T16:33:54.911 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.911 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So= 2024-03-04T16:33:54.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So=', 'Content-Length': '0'} 2024-03-04T16:33:54.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf2429d161ddb005-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.914 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.915 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.915 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.915 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.915 DEBUG:boto:Method: GET 2024-03-04T16:33:54.915 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.916 DEBUG:boto:Data: 2024-03-04T16:33:54.916 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.916 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.916 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.917 DEBUG:boto:Params: {} 2024-03-04T16:33:54.917 DEBUG:boto:Token: None 2024-03-04T16:33:54.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.917 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So= 2024-03-04T16:33:54.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So=', 'Content-Length': '0'} 2024-03-04T16:33:54.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fdd356a68485936-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.960 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.960 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.960 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.960 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.961 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.961 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.961 DEBUG:boto:Data: 2024-03-04T16:33:54.961 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.961 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.961 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.961 DEBUG:boto:Params: {} 2024-03-04T16:33:54.961 DEBUG:boto:Token: None 2024-03-04T16:33:54.961 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.961 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mPF8LPw+E6XelmdG+KpGV2VikwI= 2024-03-04T16:33:54.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mPF8LPw+E6XelmdG+KpGV2VikwI=', 'Content-Length': '0'} 2024-03-04T16:33:54.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f217044fc1e141d-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.964 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.964 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.964 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.964 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.964 DEBUG:boto:Method: HEAD 2024-03-04T16:33:54.965 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.965 DEBUG:boto:Data: 2024-03-04T16:33:54.965 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.965 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.965 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.966 DEBUG:boto:Params: {} 2024-03-04T16:33:54.966 DEBUG:boto:Token: None 2024-03-04T16:33:54.966 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.966 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mPF8LPw+E6XelmdG+KpGV2VikwI= 2024-03-04T16:33:54.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mPF8LPw+E6XelmdG+KpGV2VikwI=', 'Content-Length': '0'} 2024-03-04T16:33:54.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b6ee7622e78b4f3-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:54.969 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.969 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.970 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.970 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.970 DEBUG:boto:Method: GET 2024-03-04T16:33:54.970 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.970 DEBUG:boto:Data: 2024-03-04T16:33:54.971 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.971 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:54.971 DEBUG:boto:Port: 8001 2024-03-04T16:33:54.971 DEBUG:boto:Params: {} 2024-03-04T16:33:54.972 DEBUG:boto:Token: None 2024-03-04T16:33:54.972 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.972 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So= 2024-03-04T16:33:54.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So=', 'Content-Length': '0'} 2024-03-04T16:33:54.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000470a9d7377a4ed5a-0065e5f7f2-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:54.975 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:54.975 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:54.975 DEBUG:boto:path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.975 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.976 DEBUG:boto:Method: GET 2024-03-04T16:33:54.976 DEBUG:boto:Path: /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.976 DEBUG:boto:Data: 2024-03-04T16:33:54.976 DEBUG:boto:Headers: {} 2024-03-04T16:33:54.976 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:54.977 DEBUG:boto:Port: 8000 2024-03-04T16:33:54.977 DEBUG:boto:Params: {} 2024-03-04T16:33:54.977 DEBUG:boto:Token: None 2024-03-04T16:33:54.977 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:54 GMT /sbnvju-22/obj?versionId=BiNnWumILmRGRqS4nbXjGGtEQEPDuu2 2024-03-04T16:33:54.977 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So= 2024-03-04T16:33:54.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:54 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SelUY9LbGu4lzlVt/mco5CYh3So=', 'Content-Length': '0'} 2024-03-04T16:33:54.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'BiNnWumILmRGRqS4nbXjGGtEQEPDuu2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000738e4269331cedc5-0065e5f7f2-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.024 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.024 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.024 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.024 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.025 DEBUG:boto:Method: GET 2024-03-04T16:33:55.025 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.025 DEBUG:boto:Data: 2024-03-04T16:33:55.025 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.025 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.025 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.025 DEBUG:boto:Params: {} 2024-03-04T16:33:55.025 DEBUG:boto:Token: None 2024-03-04T16:33:55.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd60a617b2f220f9-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.029 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.029 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.029 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.029 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.029 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.029 DEBUG:boto:Method: GET 2024-03-04T16:33:55.029 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.030 DEBUG:boto:Data: 2024-03-04T16:33:55.030 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.030 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.030 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.030 DEBUG:boto:Params: {} 2024-03-04T16:33:55.030 DEBUG:boto:Token: None 2024-03-04T16:33:55.030 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58f885cce487dd7-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.033 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.033 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.033 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.034 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.034 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.034 DEBUG:boto:Data: 2024-03-04T16:33:55.034 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.034 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.034 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.034 DEBUG:boto:Params: {} 2024-03-04T16:33:55.034 DEBUG:boto:Token: None 2024-03-04T16:33:55.034 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.034 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f99ffc8f06933c2-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.038 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.038 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.038 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.038 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.039 DEBUG:boto:Data: 2024-03-04T16:33:55.039 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.039 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.039 DEBUG:boto:Params: {} 2024-03-04T16:33:55.039 DEBUG:boto:Token: None 2024-03-04T16:33:55.039 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.040 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd0425e4629a4455-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.043 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.043 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.043 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.043 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.043 DEBUG:boto:Method: GET 2024-03-04T16:33:55.043 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.043 DEBUG:boto:Data: 2024-03-04T16:33:55.043 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.043 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.044 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.044 DEBUG:boto:Params: {} 2024-03-04T16:33:55.044 DEBUG:boto:Token: None 2024-03-04T16:33:55.044 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.044 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8066873594fc3c7-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.092 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.092 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.092 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.092 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.093 DEBUG:boto:Method: GET 2024-03-04T16:33:55.093 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.093 DEBUG:boto:Data: 2024-03-04T16:33:55.093 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.093 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.093 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.093 DEBUG:boto:Params: {} 2024-03-04T16:33:55.093 DEBUG:boto:Token: None 2024-03-04T16:33:55.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.094 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008899a38a1b1b3b8c-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.096 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.096 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.096 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.096 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.096 DEBUG:boto:Method: GET 2024-03-04T16:33:55.096 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.096 DEBUG:boto:Data: 2024-03-04T16:33:55.096 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.096 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.096 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.096 DEBUG:boto:Params: {} 2024-03-04T16:33:55.097 DEBUG:boto:Token: None 2024-03-04T16:33:55.097 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.097 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580662dfb67295a3-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.100 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.100 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.101 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.101 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.101 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.101 DEBUG:boto:Method: GET 2024-03-04T16:33:55.101 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.101 DEBUG:boto:Data: 2024-03-04T16:33:55.101 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.101 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.101 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.101 DEBUG:boto:Params: {} 2024-03-04T16:33:55.101 DEBUG:boto:Token: None 2024-03-04T16:33:55.102 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.102 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd2c8cc79aee2b1c-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.104 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.104 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.105 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.105 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.105 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.105 DEBUG:boto:Method: GET 2024-03-04T16:33:55.105 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.105 DEBUG:boto:Data: 2024-03-04T16:33:55.105 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.105 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.105 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.105 DEBUG:boto:Params: {} 2024-03-04T16:33:55.105 DEBUG:boto:Token: None 2024-03-04T16:33:55.105 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.106 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc= 2024-03-04T16:33:55.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc=', 'Content-Length': '0'} 2024-03-04T16:33:55.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000315754bdf39c6ed4-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.148 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.148 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.148 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.148 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.149 DEBUG:boto:Method: GET 2024-03-04T16:33:55.149 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.149 DEBUG:boto:Data: 2024-03-04T16:33:55.149 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.149 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.149 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.149 DEBUG:boto:Params: {} 2024-03-04T16:33:55.149 DEBUG:boto:Token: None 2024-03-04T16:33:55.149 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.149 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc= 2024-03-04T16:33:55.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc=', 'Content-Length': '0'} 2024-03-04T16:33:55.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c883263d1280b69-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.196 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.196 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.197 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.197 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.197 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.197 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.197 DEBUG:boto:Data: 2024-03-04T16:33:55.197 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.197 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.197 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.197 DEBUG:boto:Params: {} 2024-03-04T16:33:55.197 DEBUG:boto:Token: None 2024-03-04T16:33:55.198 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.198 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tqggh7lC6R8tqAYFXs4at4t0w1M= 2024-03-04T16:33:55.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tqggh7lC6R8tqAYFXs4at4t0w1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a4a9fcb29596b12-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.200 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.200 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.200 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.200 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.200 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.200 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.200 DEBUG:boto:Data: 2024-03-04T16:33:55.200 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.201 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.201 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.201 DEBUG:boto:Params: {} 2024-03-04T16:33:55.201 DEBUG:boto:Token: None 2024-03-04T16:33:55.201 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.201 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:tqggh7lC6R8tqAYFXs4at4t0w1M= 2024-03-04T16:33:55.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tqggh7lC6R8tqAYFXs4at4t0w1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000180e5ff2cf416f91-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.203 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.203 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.204 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.204 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.204 DEBUG:boto:Method: GET 2024-03-04T16:33:55.204 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.204 DEBUG:boto:Data: 2024-03-04T16:33:55.204 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.204 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.204 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.204 DEBUG:boto:Params: {} 2024-03-04T16:33:55.204 DEBUG:boto:Token: None 2024-03-04T16:33:55.205 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.205 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc= 2024-03-04T16:33:55.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc=', 'Content-Length': '0'} 2024-03-04T16:33:55.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4afed494149fa46-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.248 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.248 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.248 DEBUG:boto:path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.248 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.248 DEBUG:boto:Method: GET 2024-03-04T16:33:55.249 DEBUG:boto:Path: /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.249 DEBUG:boto:Data: 2024-03-04T16:33:55.249 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.249 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.249 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.249 DEBUG:boto:Params: {} 2024-03-04T16:33:55.249 DEBUG:boto:Token: None 2024-03-04T16:33:55.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=H8LlsoTnXLqPNb1-opllLInHMzeoYsQ 2024-03-04T16:33:55.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc= 2024-03-04T16:33:55.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lMuypVaRdUQiwdcSKprPgtguDVc=', 'Content-Length': '0'} 2024-03-04T16:33:55.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'H8LlsoTnXLqPNb1-opllLInHMzeoYsQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058cd7e1fd2f236ef-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.292 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.292 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.292 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.292 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.293 DEBUG:boto:Method: GET 2024-03-04T16:33:55.293 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.293 DEBUG:boto:Data: 2024-03-04T16:33:55.293 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.293 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.293 DEBUG:boto:Params: {} 2024-03-04T16:33:55.293 DEBUG:boto:Token: None 2024-03-04T16:33:55.293 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.293 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d3d3e794ac05a3-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.296 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.297 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.297 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.297 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.297 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.297 DEBUG:boto:Method: GET 2024-03-04T16:33:55.297 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.297 DEBUG:boto:Data: 2024-03-04T16:33:55.297 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.298 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.298 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.298 DEBUG:boto:Params: {} 2024-03-04T16:33:55.298 DEBUG:boto:Token: None 2024-03-04T16:33:55.298 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000945aa3ecc4180d39-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.301 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.301 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.301 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.301 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.301 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.302 DEBUG:boto:Data: 2024-03-04T16:33:55.302 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.302 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.302 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.302 DEBUG:boto:Params: {} 2024-03-04T16:33:55.302 DEBUG:boto:Token: None 2024-03-04T16:33:55.302 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.302 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067c608b7d5b61e3a-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.305 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.306 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.306 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.306 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.306 DEBUG:boto:Data: 2024-03-04T16:33:55.306 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.306 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.306 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.306 DEBUG:boto:Params: {} 2024-03-04T16:33:55.306 DEBUG:boto:Token: None 2024-03-04T16:33:55.306 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ff30595f81dbe8e-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.310 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.310 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.310 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.310 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.310 DEBUG:boto:Method: GET 2024-03-04T16:33:55.310 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.310 DEBUG:boto:Data: 2024-03-04T16:33:55.310 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.311 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.311 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.311 DEBUG:boto:Params: {} 2024-03-04T16:33:55.311 DEBUG:boto:Token: None 2024-03-04T16:33:55.311 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.311 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052f01b395757a6ca-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.360 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.360 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.360 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.360 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.360 DEBUG:boto:Method: GET 2024-03-04T16:33:55.361 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.361 DEBUG:boto:Data: 2024-03-04T16:33:55.361 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.361 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.361 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.361 DEBUG:boto:Params: {} 2024-03-04T16:33:55.361 DEBUG:boto:Token: None 2024-03-04T16:33:55.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045829fbaef802af3-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.364 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.364 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.364 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.364 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.364 DEBUG:boto:Method: GET 2024-03-04T16:33:55.364 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.364 DEBUG:boto:Data: 2024-03-04T16:33:55.364 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.365 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.365 DEBUG:boto:Params: {} 2024-03-04T16:33:55.365 DEBUG:boto:Token: None 2024-03-04T16:33:55.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0a991c3d6d48e8-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.408 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.409 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.409 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.409 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.409 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.409 DEBUG:boto:Method: GET 2024-03-04T16:33:55.409 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.409 DEBUG:boto:Data: 2024-03-04T16:33:55.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.409 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.409 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.410 DEBUG:boto:Params: {} 2024-03-04T16:33:55.410 DEBUG:boto:Token: None 2024-03-04T16:33:55.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688edd045466a2a5-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.413 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.413 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.414 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.414 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.414 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.414 DEBUG:boto:Method: GET 2024-03-04T16:33:55.414 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.414 DEBUG:boto:Data: 2024-03-04T16:33:55.414 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.414 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.414 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.414 DEBUG:boto:Params: {} 2024-03-04T16:33:55.414 DEBUG:boto:Token: None 2024-03-04T16:33:55.415 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M= 2024-03-04T16:33:55.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c67136f74c018b3-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.464 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.464 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.464 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.465 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.465 DEBUG:boto:Method: GET 2024-03-04T16:33:55.465 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.465 DEBUG:boto:Data: 2024-03-04T16:33:55.465 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.465 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.465 DEBUG:boto:Params: {} 2024-03-04T16:33:55.465 DEBUG:boto:Token: None 2024-03-04T16:33:55.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.466 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M= 2024-03-04T16:33:55.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed5ef219b358cd7c-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.508 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.508 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.508 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.508 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.509 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.509 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.509 DEBUG:boto:Data: 2024-03-04T16:33:55.509 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.509 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.509 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.509 DEBUG:boto:Params: {} 2024-03-04T16:33:55.509 DEBUG:boto:Token: None 2024-03-04T16:33:55.509 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FRAe4RL798/Oj++f+EqyVRB8TxA= 2024-03-04T16:33:55.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FRAe4RL798/Oj++f+EqyVRB8TxA=', 'Content-Length': '0'} 2024-03-04T16:33:55.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001434f3ddf0196e8c-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.512 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.512 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.512 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.512 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.512 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.512 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.512 DEBUG:boto:Data: 2024-03-04T16:33:55.513 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.513 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.513 DEBUG:boto:Params: {} 2024-03-04T16:33:55.513 DEBUG:boto:Token: None 2024-03-04T16:33:55.513 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FRAe4RL798/Oj++f+EqyVRB8TxA= 2024-03-04T16:33:55.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FRAe4RL798/Oj++f+EqyVRB8TxA=', 'Content-Length': '0'} 2024-03-04T16:33:55.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000509f275cbf27512d-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.515 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.515 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.515 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.516 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.516 DEBUG:boto:Method: GET 2024-03-04T16:33:55.516 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.516 DEBUG:boto:Data: 2024-03-04T16:33:55.516 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.516 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.516 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.516 DEBUG:boto:Params: {} 2024-03-04T16:33:55.516 DEBUG:boto:Token: None 2024-03-04T16:33:55.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.516 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M= 2024-03-04T16:33:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009674799897e69573-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.560 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.560 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.560 DEBUG:boto:path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.561 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.561 DEBUG:boto:Method: GET 2024-03-04T16:33:55.561 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.561 DEBUG:boto:Data: 2024-03-04T16:33:55.561 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.561 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.561 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.562 DEBUG:boto:Params: {} 2024-03-04T16:33:55.562 DEBUG:boto:Token: None 2024-03-04T16:33:55.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS 2024-03-04T16:33:55.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M= 2024-03-04T16:33:55.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:geUKgvqyvo3AhSQwd3w/KhBIR1M=', 'Content-Length': '0'} 2024-03-04T16:33:55.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Cb3ZX3-Y-WJruN2bPZUe.OAPDHTN9gS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ba62c9206da0366-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.608 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.608 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.609 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.609 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.609 DEBUG:boto:Method: GET 2024-03-04T16:33:55.609 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.609 DEBUG:boto:Data: 2024-03-04T16:33:55.609 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.609 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.609 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.609 DEBUG:boto:Params: {} 2024-03-04T16:33:55.609 DEBUG:boto:Token: None 2024-03-04T16:33:55.609 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.610 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004ae0a92e9b84c4b-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.660 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.661 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.661 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.661 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.661 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.661 DEBUG:boto:Method: GET 2024-03-04T16:33:55.662 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.662 DEBUG:boto:Data: 2024-03-04T16:33:55.662 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.662 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.662 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.662 DEBUG:boto:Params: {} 2024-03-04T16:33:55.663 DEBUG:boto:Token: None 2024-03-04T16:33:55.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.663 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ffc9971e1a2e2e-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.666 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.667 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.667 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.667 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.667 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.667 DEBUG:boto:Data: 2024-03-04T16:33:55.667 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.667 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.667 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.667 DEBUG:boto:Params: {} 2024-03-04T16:33:55.667 DEBUG:boto:Token: None 2024-03-04T16:33:55.667 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.668 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d75a093952e41c7-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.671 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.671 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.672 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.672 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.672 DEBUG:boto:Data: 2024-03-04T16:33:55.672 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.672 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.672 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.672 DEBUG:boto:Params: {} 2024-03-04T16:33:55.672 DEBUG:boto:Token: None 2024-03-04T16:33:55.672 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.672 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090a3a50d8cc5fe4e-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.676 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.676 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.676 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.677 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.677 DEBUG:boto:Method: GET 2024-03-04T16:33:55.677 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.677 DEBUG:boto:Data: 2024-03-04T16:33:55.677 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.677 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.677 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.677 DEBUG:boto:Params: {} 2024-03-04T16:33:55.677 DEBUG:boto:Token: None 2024-03-04T16:33:55.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058ac18f83287c2e2-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.720 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.720 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.720 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.720 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.720 DEBUG:boto:Method: GET 2024-03-04T16:33:55.721 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.721 DEBUG:boto:Data: 2024-03-04T16:33:55.721 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.721 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.721 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.721 DEBUG:boto:Params: {} 2024-03-04T16:33:55.721 DEBUG:boto:Token: None 2024-03-04T16:33:55.721 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000725fe0e89fcfba44-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.724 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.724 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.724 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.725 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.725 DEBUG:boto:Method: GET 2024-03-04T16:33:55.725 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.725 DEBUG:boto:Data: 2024-03-04T16:33:55.725 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.725 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.725 DEBUG:boto:Params: {} 2024-03-04T16:33:55.725 DEBUG:boto:Token: None 2024-03-04T16:33:55.725 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008abb3f9bebaf90f0-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.728 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.729 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.729 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.729 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.729 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.729 DEBUG:boto:Method: GET 2024-03-04T16:33:55.729 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.729 DEBUG:boto:Data: 2024-03-04T16:33:55.729 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.729 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.730 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.730 DEBUG:boto:Params: {} 2024-03-04T16:33:55.730 DEBUG:boto:Token: None 2024-03-04T16:33:55.730 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070fd28bab506f18c-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.733 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.733 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.733 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.734 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.734 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.734 DEBUG:boto:Method: GET 2024-03-04T16:33:55.734 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.734 DEBUG:boto:Data: 2024-03-04T16:33:55.734 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.734 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.734 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.734 DEBUG:boto:Params: {} 2024-03-04T16:33:55.734 DEBUG:boto:Token: None 2024-03-04T16:33:55.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.735 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60= 2024-03-04T16:33:55.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60=', 'Content-Length': '0'} 2024-03-04T16:33:55.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0e10fd32c2a3e33-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.780 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.781 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.781 DEBUG:boto:Method: GET 2024-03-04T16:33:55.781 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.781 DEBUG:boto:Data: 2024-03-04T16:33:55.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.781 DEBUG:boto:Params: {} 2024-03-04T16:33:55.781 DEBUG:boto:Token: None 2024-03-04T16:33:55.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60= 2024-03-04T16:33:55.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60=', 'Content-Length': '0'} 2024-03-04T16:33:55.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c13ec5107aff7d54-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.784 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.785 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.785 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.785 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.785 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.786 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.786 DEBUG:boto:Data: 2024-03-04T16:33:55.786 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.786 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.786 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.786 DEBUG:boto:Params: {} 2024-03-04T16:33:55.787 DEBUG:boto:Token: None 2024-03-04T16:33:55.787 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.787 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dHnBCcDzpa3+jlNp6/OMvbOAOr0= 2024-03-04T16:33:55.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dHnBCcDzpa3+jlNp6/OMvbOAOr0=', 'Content-Length': '0'} 2024-03-04T16:33:55.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff170b708787f975-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.792 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.792 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.792 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.793 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.793 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.793 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.793 DEBUG:boto:Data: 2024-03-04T16:33:55.793 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.793 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.794 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.794 DEBUG:boto:Params: {} 2024-03-04T16:33:55.794 DEBUG:boto:Token: None 2024-03-04T16:33:55.795 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dHnBCcDzpa3+jlNp6/OMvbOAOr0= 2024-03-04T16:33:55.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dHnBCcDzpa3+jlNp6/OMvbOAOr0=', 'Content-Length': '0'} 2024-03-04T16:33:55.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6fb203046129657-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.798 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.798 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.798 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.798 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.799 DEBUG:boto:Method: GET 2024-03-04T16:33:55.799 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.799 DEBUG:boto:Data: 2024-03-04T16:33:55.799 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.799 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.799 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.799 DEBUG:boto:Params: {} 2024-03-04T16:33:55.799 DEBUG:boto:Token: None 2024-03-04T16:33:55.799 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.799 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60= 2024-03-04T16:33:55.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60=', 'Content-Length': '0'} 2024-03-04T16:33:55.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a93eaf01ffbba67-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.848 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.848 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.848 DEBUG:boto:path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.848 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.849 DEBUG:boto:Method: GET 2024-03-04T16:33:55.849 DEBUG:boto:Path: /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.849 DEBUG:boto:Data: 2024-03-04T16:33:55.849 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.849 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.849 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.849 DEBUG:boto:Params: {} 2024-03-04T16:33:55.849 DEBUG:boto:Token: None 2024-03-04T16:33:55.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=XiCW4ml6JGIKsAt..485xm6JEN33ZeJ 2024-03-04T16:33:55.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60= 2024-03-04T16:33:55.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C79JRBlp5the689Jh7jsI9m4n60=', 'Content-Length': '0'} 2024-03-04T16:33:55.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'XiCW4ml6JGIKsAt..485xm6JEN33ZeJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000125c02f8e41e9b75-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.852 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.852 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.852 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.853 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.853 DEBUG:boto:Method: GET 2024-03-04T16:33:55.853 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.853 DEBUG:boto:Data: 2024-03-04T16:33:55.853 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.853 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.853 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.853 DEBUG:boto:Params: {} 2024-03-04T16:33:55.853 DEBUG:boto:Token: None 2024-03-04T16:33:55.854 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.854 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d87cbd3a6a62da-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.857 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.857 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.857 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.857 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.858 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.858 DEBUG:boto:Method: GET 2024-03-04T16:33:55.858 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.858 DEBUG:boto:Data: 2024-03-04T16:33:55.858 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.858 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.858 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.858 DEBUG:boto:Params: {} 2024-03-04T16:33:55.858 DEBUG:boto:Token: None 2024-03-04T16:33:55.859 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.859 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090eeca1d1675a396-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.908 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.909 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.909 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.909 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.909 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.909 DEBUG:boto:Data: 2024-03-04T16:33:55.909 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.909 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.910 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.910 DEBUG:boto:Params: {} 2024-03-04T16:33:55.910 DEBUG:boto:Token: None 2024-03-04T16:33:55.910 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba45f6a7a57410d2-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.914 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.914 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.914 DEBUG:boto:Method: HEAD 2024-03-04T16:33:55.915 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:55.915 DEBUG:boto:Data: 2024-03-04T16:33:55.915 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.915 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.915 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.915 DEBUG:boto:Params: {} 2024-03-04T16:33:55.915 DEBUG:boto:Token: None 2024-03-04T16:33:55.915 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj 2024-03-04T16:33:55.915 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U= 2024-03-04T16:33:55.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c8lxqI5AdpAx9qunP1Nl+k/dG4U=', 'Content-Length': '0'} 2024-03-04T16:33:55.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021c517969e0b7af0-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.919 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.919 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.919 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.919 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.919 DEBUG:boto:Method: GET 2024-03-04T16:33:55.920 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.920 DEBUG:boto:Data: 2024-03-04T16:33:55.920 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.920 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.920 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.920 DEBUG:boto:Params: {} 2024-03-04T16:33:55.920 DEBUG:boto:Token: None 2024-03-04T16:33:55.920 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.921 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ae17279a2ca246-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.968 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.968 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.968 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.968 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.969 DEBUG:boto:Method: GET 2024-03-04T16:33:55.969 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.969 DEBUG:boto:Data: 2024-03-04T16:33:55.969 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.969 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.969 DEBUG:boto:Params: {} 2024-03-04T16:33:55.969 DEBUG:boto:Token: None 2024-03-04T16:33:55.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:55.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw= 2024-03-04T16:33:55.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m3PFVE592Dr/yhxZYZpWqIMWxtw=', 'Content-Length': '0'} 2024-03-04T16:33:55.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b47a23f4abb7487c-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.972 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.972 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.972 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.972 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.972 DEBUG:boto:Method: GET 2024-03-04T16:33:55.972 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.973 DEBUG:boto:Data: 2024-03-04T16:33:55.973 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.973 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.973 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.973 DEBUG:boto:Params: {} 2024-03-04T16:33:55.973 DEBUG:boto:Token: None 2024-03-04T16:33:55.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc9deb915561bef-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.977 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.978 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:55.978 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:55.978 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.978 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:55.978 DEBUG:boto:Method: GET 2024-03-04T16:33:55.978 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:55.978 DEBUG:boto:Data: 2024-03-04T16:33:55.979 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.979 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.979 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.979 DEBUG:boto:Params: {} 2024-03-04T16:33:55.979 DEBUG:boto:Token: None 2024-03-04T16:33:55.979 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/ 2024-03-04T16:33:55.979 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0= 2024-03-04T16:33:55.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qIOH8PCbFNovQWyXB0HjayXH6n0=', 'Content-Length': '0'} 2024-03-04T16:33:55.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385302afc25f93f2-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.982 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:55.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:55.983 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.983 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.983 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.983 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.983 DEBUG:boto:Method: GET 2024-03-04T16:33:55.983 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.983 DEBUG:boto:Data: 2024-03-04T16:33:55.983 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.983 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:55.983 DEBUG:boto:Port: 8001 2024-03-04T16:33:55.984 DEBUG:boto:Params: {} 2024-03-04T16:33:55.984 DEBUG:boto:Token: None 2024-03-04T16:33:55.984 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.984 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DYVdFGPV11vnWWts3vAtRNbI9OA= 2024-03-04T16:33:55.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DYVdFGPV11vnWWts3vAtRNbI9OA=', 'Content-Length': '0'} 2024-03-04T16:33:55.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000258793ccb9d30867-0065e5f7f3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:55.987 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:55.987 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:55.987 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.987 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.987 DEBUG:boto:Method: GET 2024-03-04T16:33:55.987 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.987 DEBUG:boto:Data: 2024-03-04T16:33:55.987 DEBUG:boto:Headers: {} 2024-03-04T16:33:55.988 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:55.988 DEBUG:boto:Port: 8000 2024-03-04T16:33:55.988 DEBUG:boto:Params: {} 2024-03-04T16:33:55.988 DEBUG:boto:Token: None 2024-03-04T16:33:55.988 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:55 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:55.988 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DYVdFGPV11vnWWts3vAtRNbI9OA= 2024-03-04T16:33:55.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DYVdFGPV11vnWWts3vAtRNbI9OA=', 'Content-Length': '0'} 2024-03-04T16:33:55.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f7aed75010df1f2-0065e5f7f3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.036 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.036 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.036 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.037 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.037 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.037 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.037 DEBUG:boto:Data: 2024-03-04T16:33:56.037 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.037 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.037 DEBUG:boto:Params: {} 2024-03-04T16:33:56.037 DEBUG:boto:Token: None 2024-03-04T16:33:56.038 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A7UodT2KnfhrPTBegy4uL/T4+ck= 2024-03-04T16:33:56.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A7UodT2KnfhrPTBegy4uL/T4+ck=', 'Content-Length': '0'} 2024-03-04T16:33:56.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1013f3c8dbba3dc-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.040 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.040 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.040 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.040 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.040 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.041 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.041 DEBUG:boto:Data: 2024-03-04T16:33:56.041 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.041 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.041 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.041 DEBUG:boto:Params: {} 2024-03-04T16:33:56.041 DEBUG:boto:Token: None 2024-03-04T16:33:56.041 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.041 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:A7UodT2KnfhrPTBegy4uL/T4+ck= 2024-03-04T16:33:56.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A7UodT2KnfhrPTBegy4uL/T4+ck=', 'Content-Length': '0'} 2024-03-04T16:33:56.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000449b6d63dde8d63a-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.044 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.044 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.044 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.044 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.044 DEBUG:boto:Method: GET 2024-03-04T16:33:56.044 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.044 DEBUG:boto:Data: 2024-03-04T16:33:56.044 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.045 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.045 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.045 DEBUG:boto:Params: {} 2024-03-04T16:33:56.045 DEBUG:boto:Token: None 2024-03-04T16:33:56.045 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.045 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4Nl1s1Hpp9OMC1sJpjWe82o6iK0= 2024-03-04T16:33:56.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4Nl1s1Hpp9OMC1sJpjWe82o6iK0=', 'Content-Length': '0'} 2024-03-04T16:33:56.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea2c0282b605438e-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.047 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.048 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.048 DEBUG:boto:path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.048 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.048 DEBUG:boto:Method: GET 2024-03-04T16:33:56.048 DEBUG:boto:Path: /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.048 DEBUG:boto:Data: 2024-03-04T16:33:56.048 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.048 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.048 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.048 DEBUG:boto:Params: {} 2024-03-04T16:33:56.048 DEBUG:boto:Token: None 2024-03-04T16:33:56.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk 2024-03-04T16:33:56.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4Nl1s1Hpp9OMC1sJpjWe82o6iK0= 2024-03-04T16:33:56.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4Nl1s1Hpp9OMC1sJpjWe82o6iK0=', 'Content-Length': '0'} 2024-03-04T16:33:56.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', 'f9y2fO3FZIWX9Lki2lP3yXMrAmJcFhk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004918e80fdff49633-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.092 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.092 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.093 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.093 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.093 DEBUG:boto:Method: GET 2024-03-04T16:33:56.093 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.093 DEBUG:boto:Data: 2024-03-04T16:33:56.093 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.093 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.093 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.093 DEBUG:boto:Params: {} 2024-03-04T16:33:56.093 DEBUG:boto:Token: None 2024-03-04T16:33:56.093 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.094 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039347391e04ddc2d-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.097 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.097 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.097 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.097 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.097 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.098 DEBUG:boto:Method: GET 2024-03-04T16:33:56.098 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.098 DEBUG:boto:Data: 2024-03-04T16:33:56.098 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.098 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.098 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.098 DEBUG:boto:Params: {} 2024-03-04T16:33:56.098 DEBUG:boto:Token: None 2024-03-04T16:33:56.098 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.098 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712209ea9eeb9992-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.148 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.148 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.149 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.149 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.149 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.149 DEBUG:boto:Data: 2024-03-04T16:33:56.149 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.149 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.149 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.149 DEBUG:boto:Params: {} 2024-03-04T16:33:56.149 DEBUG:boto:Token: None 2024-03-04T16:33:56.150 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.150 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3fe4d2288fda8e0-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.153 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.154 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.154 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.154 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.154 DEBUG:boto:Data: 2024-03-04T16:33:56.154 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.154 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.154 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.154 DEBUG:boto:Params: {} 2024-03-04T16:33:56.154 DEBUG:boto:Token: None 2024-03-04T16:33:56.155 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.155 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec45c7dd0175a15f-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.158 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.158 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.158 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.159 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.159 DEBUG:boto:Method: GET 2024-03-04T16:33:56.159 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.159 DEBUG:boto:Data: 2024-03-04T16:33:56.159 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.159 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.159 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.159 DEBUG:boto:Params: {} 2024-03-04T16:33:56.159 DEBUG:boto:Token: None 2024-03-04T16:33:56.159 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec3a85303b15d451-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.204 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.205 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.205 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.205 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.205 DEBUG:boto:Method: GET 2024-03-04T16:33:56.205 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.205 DEBUG:boto:Data: 2024-03-04T16:33:56.205 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.205 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.205 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.205 DEBUG:boto:Params: {} 2024-03-04T16:33:56.205 DEBUG:boto:Token: None 2024-03-04T16:33:56.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.206 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddd33eb7b82bbd36-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.209 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.209 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.209 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.209 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.209 DEBUG:boto:Method: GET 2024-03-04T16:33:56.209 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.210 DEBUG:boto:Data: 2024-03-04T16:33:56.210 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.210 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.210 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.210 DEBUG:boto:Params: {} 2024-03-04T16:33:56.210 DEBUG:boto:Token: None 2024-03-04T16:33:56.210 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.211 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebd7ea00a92ba9b1-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.214 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.214 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.214 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.214 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.214 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.214 DEBUG:boto:Method: GET 2024-03-04T16:33:56.215 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.215 DEBUG:boto:Data: 2024-03-04T16:33:56.215 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.215 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.215 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.215 DEBUG:boto:Params: {} 2024-03-04T16:33:56.215 DEBUG:boto:Token: None 2024-03-04T16:33:56.215 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.216 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024aa8dc95fdb8001-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.219 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.219 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.219 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.220 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.220 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.221 DEBUG:boto:Method: GET 2024-03-04T16:33:56.221 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.221 DEBUG:boto:Data: 2024-03-04T16:33:56.221 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.221 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.221 DEBUG:boto:Params: {} 2024-03-04T16:33:56.222 DEBUG:boto:Token: None 2024-03-04T16:33:56.222 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08= 2024-03-04T16:33:56.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08=', 'Content-Length': '0'} 2024-03-04T16:33:56.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600dd238569316f4-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.224 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.224 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.224 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.225 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.225 DEBUG:boto:Method: GET 2024-03-04T16:33:56.225 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.225 DEBUG:boto:Data: 2024-03-04T16:33:56.225 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.225 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.225 DEBUG:boto:Params: {} 2024-03-04T16:33:56.225 DEBUG:boto:Token: None 2024-03-04T16:33:56.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08= 2024-03-04T16:33:56.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08=', 'Content-Length': '0'} 2024-03-04T16:33:56.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd2c147d5d3d87b8-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.272 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.272 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.272 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.272 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.273 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.273 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.273 DEBUG:boto:Data: 2024-03-04T16:33:56.273 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.273 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.273 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.273 DEBUG:boto:Params: {} 2024-03-04T16:33:56.273 DEBUG:boto:Token: None 2024-03-04T16:33:56.273 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.273 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0RM71jHlWPWzHUHAa33MxQwUewc= 2024-03-04T16:33:56.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0RM71jHlWPWzHUHAa33MxQwUewc=', 'Content-Length': '0'} 2024-03-04T16:33:56.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000260167adb4766aee-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.275 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.276 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.276 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.276 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.276 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.276 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.276 DEBUG:boto:Data: 2024-03-04T16:33:56.276 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.276 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.276 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.276 DEBUG:boto:Params: {} 2024-03-04T16:33:56.277 DEBUG:boto:Token: None 2024-03-04T16:33:56.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0RM71jHlWPWzHUHAa33MxQwUewc= 2024-03-04T16:33:56.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0RM71jHlWPWzHUHAa33MxQwUewc=', 'Content-Length': '0'} 2024-03-04T16:33:56.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a361df504facd6b-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.279 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.279 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.279 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.279 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.279 DEBUG:boto:Method: GET 2024-03-04T16:33:56.280 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.280 DEBUG:boto:Data: 2024-03-04T16:33:56.280 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.280 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.280 DEBUG:boto:Params: {} 2024-03-04T16:33:56.280 DEBUG:boto:Token: None 2024-03-04T16:33:56.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08= 2024-03-04T16:33:56.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08=', 'Content-Length': '0'} 2024-03-04T16:33:56.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d311f687345c02a9-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.282 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.283 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.283 DEBUG:boto:path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.283 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.283 DEBUG:boto:Method: GET 2024-03-04T16:33:56.283 DEBUG:boto:Path: /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.283 DEBUG:boto:Data: 2024-03-04T16:33:56.283 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.283 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.283 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.283 DEBUG:boto:Params: {} 2024-03-04T16:33:56.283 DEBUG:boto:Token: None 2024-03-04T16:33:56.284 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=UWMoj2KG9hGrAO1d83MedK1E8JGdxkK 2024-03-04T16:33:56.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08= 2024-03-04T16:33:56.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NbAfq+eOsgCqGJG1fLcs+E/NG08=', 'Content-Length': '0'} 2024-03-04T16:33:56.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'UWMoj2KG9hGrAO1d83MedK1E8JGdxkK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000021c881135e04bba-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.332 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.332 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.332 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.332 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.333 DEBUG:boto:Method: GET 2024-03-04T16:33:56.333 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.333 DEBUG:boto:Data: 2024-03-04T16:33:56.333 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.333 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.333 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.333 DEBUG:boto:Params: {} 2024-03-04T16:33:56.333 DEBUG:boto:Token: None 2024-03-04T16:33:56.333 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.333 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4fbd8e670e6f0f-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.337 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.337 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.337 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.337 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.337 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.337 DEBUG:boto:Method: GET 2024-03-04T16:33:56.337 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.338 DEBUG:boto:Data: 2024-03-04T16:33:56.338 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.338 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.338 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.338 DEBUG:boto:Params: {} 2024-03-04T16:33:56.338 DEBUG:boto:Token: None 2024-03-04T16:33:56.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c89820ffd4e1722-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.341 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.341 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.342 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.342 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.342 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.342 DEBUG:boto:Data: 2024-03-04T16:33:56.342 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.342 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.342 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.342 DEBUG:boto:Params: {} 2024-03-04T16:33:56.342 DEBUG:boto:Token: None 2024-03-04T16:33:56.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.343 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b35d326409bc8213-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.346 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.346 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.347 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.347 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.347 DEBUG:boto:Data: 2024-03-04T16:33:56.347 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.347 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.347 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.347 DEBUG:boto:Params: {} 2024-03-04T16:33:56.347 DEBUG:boto:Token: None 2024-03-04T16:33:56.348 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.348 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1a41aedd33b5ccb-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.352 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.352 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.352 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.352 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.352 DEBUG:boto:Method: GET 2024-03-04T16:33:56.353 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.353 DEBUG:boto:Data: 2024-03-04T16:33:56.353 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.353 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.353 DEBUG:boto:Params: {} 2024-03-04T16:33:56.353 DEBUG:boto:Token: None 2024-03-04T16:33:56.353 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d41553f63848fb3-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.397 DEBUG:boto:Method: GET 2024-03-04T16:33:56.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.397 DEBUG:boto:Data: 2024-03-04T16:33:56.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.397 DEBUG:boto:Params: {} 2024-03-04T16:33:56.397 DEBUG:boto:Token: None 2024-03-04T16:33:56.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040b5024c5480d6ef-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.400 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.400 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.400 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.400 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.400 DEBUG:boto:Method: GET 2024-03-04T16:33:56.400 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.400 DEBUG:boto:Data: 2024-03-04T16:33:56.400 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.401 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.401 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.401 DEBUG:boto:Params: {} 2024-03-04T16:33:56.401 DEBUG:boto:Token: None 2024-03-04T16:33:56.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6f08d5279dcda2f-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.404 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.404 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.405 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.405 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.405 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.405 DEBUG:boto:Method: GET 2024-03-04T16:33:56.405 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.405 DEBUG:boto:Data: 2024-03-04T16:33:56.405 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.405 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.405 DEBUG:boto:Params: {} 2024-03-04T16:33:56.405 DEBUG:boto:Token: None 2024-03-04T16:33:56.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef921120ff078ae-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.408 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.409 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.409 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.409 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.409 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.409 DEBUG:boto:Method: GET 2024-03-04T16:33:56.409 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.409 DEBUG:boto:Data: 2024-03-04T16:33:56.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.409 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.410 DEBUG:boto:Params: {} 2024-03-04T16:33:56.410 DEBUG:boto:Token: None 2024-03-04T16:33:56.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I= 2024-03-04T16:33:56.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I=', 'Content-Length': '0'} 2024-03-04T16:33:56.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f5a11f2a739f1a5-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.460 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.460 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.460 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.460 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.461 DEBUG:boto:Method: GET 2024-03-04T16:33:56.461 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.461 DEBUG:boto:Data: 2024-03-04T16:33:56.461 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.461 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.461 DEBUG:boto:Params: {} 2024-03-04T16:33:56.461 DEBUG:boto:Token: None 2024-03-04T16:33:56.461 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I= 2024-03-04T16:33:56.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I=', 'Content-Length': '0'} 2024-03-04T16:33:56.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097cc3edc6537687c-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.504 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.504 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.504 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.504 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.505 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.505 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.505 DEBUG:boto:Data: 2024-03-04T16:33:56.505 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.505 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.505 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.505 DEBUG:boto:Params: {} 2024-03-04T16:33:56.505 DEBUG:boto:Token: None 2024-03-04T16:33:56.505 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IyPlAhRD0eT2HbSbFHouhRoOqQA= 2024-03-04T16:33:56.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IyPlAhRD0eT2HbSbFHouhRoOqQA=', 'Content-Length': '0'} 2024-03-04T16:33:56.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075dfbc6e101a82f2-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.508 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.508 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.508 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.508 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.508 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.508 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.508 DEBUG:boto:Data: 2024-03-04T16:33:56.508 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.509 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.509 DEBUG:boto:Params: {} 2024-03-04T16:33:56.509 DEBUG:boto:Token: None 2024-03-04T16:33:56.509 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IyPlAhRD0eT2HbSbFHouhRoOqQA= 2024-03-04T16:33:56.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IyPlAhRD0eT2HbSbFHouhRoOqQA=', 'Content-Length': '0'} 2024-03-04T16:33:56.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a4cf618fc30580a-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.512 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.512 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.512 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.512 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.512 DEBUG:boto:Method: GET 2024-03-04T16:33:56.512 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.512 DEBUG:boto:Data: 2024-03-04T16:33:56.512 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.512 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.512 DEBUG:boto:Params: {} 2024-03-04T16:33:56.513 DEBUG:boto:Token: None 2024-03-04T16:33:56.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I= 2024-03-04T16:33:56.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I=', 'Content-Length': '0'} 2024-03-04T16:33:56.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fef700cc18e73c2-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.557 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.557 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.557 DEBUG:boto:path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.557 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.557 DEBUG:boto:Method: GET 2024-03-04T16:33:56.557 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.557 DEBUG:boto:Data: 2024-03-04T16:33:56.557 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.557 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.558 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.558 DEBUG:boto:Params: {} 2024-03-04T16:33:56.558 DEBUG:boto:Token: None 2024-03-04T16:33:56.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R 2024-03-04T16:33:56.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I= 2024-03-04T16:33:56.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2zr6l3tmYk4OzJiQcF9WlLqDl4I=', 'Content-Length': '0'} 2024-03-04T16:33:56.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Z1-iuDFIejtz7F1ccZKbN2mDV7h.h9R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8d770042c9c802f-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.608 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.609 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.609 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.609 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.609 DEBUG:boto:Method: GET 2024-03-04T16:33:56.609 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.612 DEBUG:boto:Data: 2024-03-04T16:33:56.612 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.612 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.612 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.612 DEBUG:boto:Params: {} 2024-03-04T16:33:56.612 DEBUG:boto:Token: None 2024-03-04T16:33:56.613 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.613 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6d9f4830343e65-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.616 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.616 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.616 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.616 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.616 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.617 DEBUG:boto:Method: GET 2024-03-04T16:33:56.617 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.617 DEBUG:boto:Data: 2024-03-04T16:33:56.617 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.617 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.617 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.617 DEBUG:boto:Params: {} 2024-03-04T16:33:56.617 DEBUG:boto:Token: None 2024-03-04T16:33:56.617 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.617 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087c57d214e2230e-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.620 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.620 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.620 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.621 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.621 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.621 DEBUG:boto:Data: 2024-03-04T16:33:56.621 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.621 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.621 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.622 DEBUG:boto:Params: {} 2024-03-04T16:33:56.622 DEBUG:boto:Token: None 2024-03-04T16:33:56.622 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057b5f595f7461fec-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.625 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.626 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.626 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.626 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.626 DEBUG:boto:Data: 2024-03-04T16:33:56.626 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.626 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.626 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.626 DEBUG:boto:Params: {} 2024-03-04T16:33:56.626 DEBUG:boto:Token: None 2024-03-04T16:33:56.626 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b704a2504668e5b0-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.630 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.630 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.630 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.630 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.630 DEBUG:boto:Method: GET 2024-03-04T16:33:56.630 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.630 DEBUG:boto:Data: 2024-03-04T16:33:56.631 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.631 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.631 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.631 DEBUG:boto:Params: {} 2024-03-04T16:33:56.631 DEBUG:boto:Token: None 2024-03-04T16:33:56.631 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.631 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a0c789ba9536fef-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.676 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.676 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.676 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.677 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.677 DEBUG:boto:Method: GET 2024-03-04T16:33:56.677 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.677 DEBUG:boto:Data: 2024-03-04T16:33:56.677 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.677 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.677 DEBUG:boto:Params: {} 2024-03-04T16:33:56.677 DEBUG:boto:Token: None 2024-03-04T16:33:56.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a68d7b6e8b1aeb2-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.680 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.681 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.681 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.681 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.681 DEBUG:boto:Method: GET 2024-03-04T16:33:56.681 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.681 DEBUG:boto:Data: 2024-03-04T16:33:56.681 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.681 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.681 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.681 DEBUG:boto:Params: {} 2024-03-04T16:33:56.681 DEBUG:boto:Token: None 2024-03-04T16:33:56.682 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.682 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cea6773aeb250c8-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.685 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.685 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.685 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.685 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.685 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.685 DEBUG:boto:Method: GET 2024-03-04T16:33:56.686 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.686 DEBUG:boto:Data: 2024-03-04T16:33:56.686 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.686 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.686 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.686 DEBUG:boto:Params: {} 2024-03-04T16:33:56.686 DEBUG:boto:Token: None 2024-03-04T16:33:56.686 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.686 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6892a55f4540de0-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.732 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.733 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.733 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.733 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.733 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.733 DEBUG:boto:Method: GET 2024-03-04T16:33:56.733 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.733 DEBUG:boto:Data: 2024-03-04T16:33:56.733 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.733 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.733 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.733 DEBUG:boto:Params: {} 2024-03-04T16:33:56.734 DEBUG:boto:Token: None 2024-03-04T16:33:56.734 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.734 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc= 2024-03-04T16:33:56.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc=', 'Content-Length': '0'} 2024-03-04T16:33:56.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a531f0e64eb80b1e-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.780 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.780 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.781 DEBUG:boto:Method: GET 2024-03-04T16:33:56.781 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.781 DEBUG:boto:Data: 2024-03-04T16:33:56.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.781 DEBUG:boto:Params: {} 2024-03-04T16:33:56.781 DEBUG:boto:Token: None 2024-03-04T16:33:56.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc= 2024-03-04T16:33:56.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc=', 'Content-Length': '0'} 2024-03-04T16:33:56.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070dfd60dab3e846c-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.784 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.784 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.784 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.784 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.784 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.784 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.785 DEBUG:boto:Data: 2024-03-04T16:33:56.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.785 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.785 DEBUG:boto:Params: {} 2024-03-04T16:33:56.785 DEBUG:boto:Token: None 2024-03-04T16:33:56.785 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S92e6UnRDnAMhxA6DXoyYZxygFY= 2024-03-04T16:33:56.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S92e6UnRDnAMhxA6DXoyYZxygFY=', 'Content-Length': '0'} 2024-03-04T16:33:56.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d224883bf0497d62-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.788 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.788 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.788 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.788 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.788 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.788 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.788 DEBUG:boto:Data: 2024-03-04T16:33:56.788 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.788 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.788 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.789 DEBUG:boto:Params: {} 2024-03-04T16:33:56.789 DEBUG:boto:Token: None 2024-03-04T16:33:56.789 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S92e6UnRDnAMhxA6DXoyYZxygFY= 2024-03-04T16:33:56.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S92e6UnRDnAMhxA6DXoyYZxygFY=', 'Content-Length': '0'} 2024-03-04T16:33:56.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cd5db3e4503c163-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.791 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.791 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.791 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.792 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.792 DEBUG:boto:Method: GET 2024-03-04T16:33:56.792 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.792 DEBUG:boto:Data: 2024-03-04T16:33:56.792 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.792 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.792 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.792 DEBUG:boto:Params: {} 2024-03-04T16:33:56.792 DEBUG:boto:Token: None 2024-03-04T16:33:56.792 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.793 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc= 2024-03-04T16:33:56.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc=', 'Content-Length': '0'} 2024-03-04T16:33:56.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf00e3d34ada7a75-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.840 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.840 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.841 DEBUG:boto:path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.841 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.841 DEBUG:boto:Method: GET 2024-03-04T16:33:56.841 DEBUG:boto:Path: /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.841 DEBUG:boto:Data: 2024-03-04T16:33:56.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.841 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.841 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.841 DEBUG:boto:Params: {} 2024-03-04T16:33:56.841 DEBUG:boto:Token: None 2024-03-04T16:33:56.842 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=ahIF6D4RiReN93jnXI2J7WYQmcubz3L 2024-03-04T16:33:56.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc= 2024-03-04T16:33:56.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:G/nyPCwzeQbVgjpEnY6mdwFIIAc=', 'Content-Length': '0'} 2024-03-04T16:33:56.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'ahIF6D4RiReN93jnXI2J7WYQmcubz3L'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0f11635441132c6-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.844 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.844 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.844 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.844 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.844 DEBUG:boto:Method: GET 2024-03-04T16:33:56.844 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.845 DEBUG:boto:Data: 2024-03-04T16:33:56.845 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.845 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.845 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.845 DEBUG:boto:Params: {} 2024-03-04T16:33:56.846 DEBUG:boto:Token: None 2024-03-04T16:33:56.846 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009dfd2a8e37faba1-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.849 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.849 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.849 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.849 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.850 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.850 DEBUG:boto:Method: GET 2024-03-04T16:33:56.850 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.850 DEBUG:boto:Data: 2024-03-04T16:33:56.850 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.850 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.850 DEBUG:boto:Params: {} 2024-03-04T16:33:56.851 DEBUG:boto:Token: None 2024-03-04T16:33:56.851 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.851 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083c2cdedc5b261d8-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.853 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.854 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.854 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.854 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.854 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.854 DEBUG:boto:Data: 2024-03-04T16:33:56.854 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.855 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.855 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.855 DEBUG:boto:Params: {} 2024-03-04T16:33:56.855 DEBUG:boto:Token: None 2024-03-04T16:33:56.855 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045618188f379da5d-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.859 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.859 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.859 DEBUG:boto:Method: HEAD 2024-03-04T16:33:56.859 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:56.859 DEBUG:boto:Data: 2024-03-04T16:33:56.859 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.860 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.860 DEBUG:boto:Params: {} 2024-03-04T16:33:56.860 DEBUG:boto:Token: None 2024-03-04T16:33:56.860 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj 2024-03-04T16:33:56.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY= 2024-03-04T16:33:56.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5WMxJg285VbE1caDUIWYpw2u2mY=', 'Content-Length': '0'} 2024-03-04T16:33:56.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ccf78a4deb7aaac-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.863 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.864 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.864 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.864 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.864 DEBUG:boto:Method: GET 2024-03-04T16:33:56.864 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.864 DEBUG:boto:Data: 2024-03-04T16:33:56.864 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.864 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.864 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.864 DEBUG:boto:Params: {} 2024-03-04T16:33:56.864 DEBUG:boto:Token: None 2024-03-04T16:33:56.865 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.865 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085c7493da617f90a-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.912 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.912 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.912 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.912 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.913 DEBUG:boto:Method: GET 2024-03-04T16:33:56.913 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.913 DEBUG:boto:Data: 2024-03-04T16:33:56.913 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.913 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.913 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.913 DEBUG:boto:Params: {} 2024-03-04T16:33:56.913 DEBUG:boto:Token: None 2024-03-04T16:33:56.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:56.914 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs= 2024-03-04T16:33:56.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YWela9LK9JFqEl8LzrZ59rpcjPs=', 'Content-Length': '0'} 2024-03-04T16:33:56.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000541256c7afb6caa5-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.916 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.916 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.916 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.916 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.916 DEBUG:boto:Method: GET 2024-03-04T16:33:56.916 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.916 DEBUG:boto:Data: 2024-03-04T16:33:56.917 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.917 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.917 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.917 DEBUG:boto:Params: {} 2024-03-04T16:33:56.917 DEBUG:boto:Token: None 2024-03-04T16:33:56.917 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.917 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3303f487c58300a-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.960 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.960 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:56.963 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:56.963 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.963 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:56.963 DEBUG:boto:Method: GET 2024-03-04T16:33:56.963 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:56.963 DEBUG:boto:Data: 2024-03-04T16:33:56.963 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.963 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.963 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.963 DEBUG:boto:Params: {} 2024-03-04T16:33:56.964 DEBUG:boto:Token: None 2024-03-04T16:33:56.964 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/ 2024-03-04T16:33:56.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk= 2024-03-04T16:33:56.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gEPkkfJvv4h8UfOx659D5FfJ3bk=', 'Content-Length': '0'} 2024-03-04T16:33:56.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54f929c41098ff8-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.967 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:56.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:56.968 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.968 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.968 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.969 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.969 DEBUG:boto:Method: GET 2024-03-04T16:33:56.969 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.969 DEBUG:boto:Data: 2024-03-04T16:33:56.969 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.969 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:56.969 DEBUG:boto:Port: 8001 2024-03-04T16:33:56.970 DEBUG:boto:Params: {} 2024-03-04T16:33:56.970 DEBUG:boto:Token: None 2024-03-04T16:33:56.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.970 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5eNwnKHJQ08U4r/1z5uMoJ/hkic= 2024-03-04T16:33:56.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5eNwnKHJQ08U4r/1z5uMoJ/hkic=', 'Content-Length': '0'} 2024-03-04T16:33:56.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7cbbb5761c4b782-0065e5f7f4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:56.972 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:56.972 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:56.973 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.973 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.973 DEBUG:boto:Method: GET 2024-03-04T16:33:56.973 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.973 DEBUG:boto:Data: 2024-03-04T16:33:56.973 DEBUG:boto:Headers: {} 2024-03-04T16:33:56.973 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:56.973 DEBUG:boto:Port: 8000 2024-03-04T16:33:56.973 DEBUG:boto:Params: {} 2024-03-04T16:33:56.974 DEBUG:boto:Token: None 2024-03-04T16:33:56.974 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:56 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:56.974 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5eNwnKHJQ08U4r/1z5uMoJ/hkic= 2024-03-04T16:33:56.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:56 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5eNwnKHJQ08U4r/1z5uMoJ/hkic=', 'Content-Length': '0'} 2024-03-04T16:33:56.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad7dc6e18323a0d4-0065e5f7f4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:56 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.016 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.016 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.016 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.016 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.017 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.017 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.017 DEBUG:boto:Data: 2024-03-04T16:33:57.017 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.017 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.017 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.017 DEBUG:boto:Params: {} 2024-03-04T16:33:57.017 DEBUG:boto:Token: None 2024-03-04T16:33:57.017 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.018 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DvaTsSjHQOCaYKTe3m51CbneehY= 2024-03-04T16:33:57.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DvaTsSjHQOCaYKTe3m51CbneehY=', 'Content-Length': '0'} 2024-03-04T16:33:57.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f305ff535167540d-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.020 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.020 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.020 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.020 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.020 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.020 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.020 DEBUG:boto:Data: 2024-03-04T16:33:57.021 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.021 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.021 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.021 DEBUG:boto:Params: {} 2024-03-04T16:33:57.021 DEBUG:boto:Token: None 2024-03-04T16:33:57.021 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.021 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DvaTsSjHQOCaYKTe3m51CbneehY= 2024-03-04T16:33:57.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DvaTsSjHQOCaYKTe3m51CbneehY=', 'Content-Length': '0'} 2024-03-04T16:33:57.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004730417545731a6f-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.024 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.024 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.024 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.024 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.024 DEBUG:boto:Method: GET 2024-03-04T16:33:57.024 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.024 DEBUG:boto:Data: 2024-03-04T16:33:57.024 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.025 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.025 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.025 DEBUG:boto:Params: {} 2024-03-04T16:33:57.025 DEBUG:boto:Token: None 2024-03-04T16:33:57.025 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.025 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AGgvc1fhDqedvpz8145uLS6AIgk= 2024-03-04T16:33:57.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AGgvc1fhDqedvpz8145uLS6AIgk=', 'Content-Length': '0'} 2024-03-04T16:33:57.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008af43803234413f0-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.027 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.028 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.028 DEBUG:boto:path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.028 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.028 DEBUG:boto:Method: GET 2024-03-04T16:33:57.028 DEBUG:boto:Path: /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.028 DEBUG:boto:Data: 2024-03-04T16:33:57.028 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.028 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.028 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.028 DEBUG:boto:Params: {} 2024-03-04T16:33:57.028 DEBUG:boto:Token: None 2024-03-04T16:33:57.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8 2024-03-04T16:33:57.029 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AGgvc1fhDqedvpz8145uLS6AIgk= 2024-03-04T16:33:57.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AGgvc1fhDqedvpz8145uLS6AIgk=', 'Content-Length': '0'} 2024-03-04T16:33:57.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'O4-mxF8QjsFbHak4Z.gx3Qyb0EbmsG8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7fd70899fdc5248-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.076 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.076 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.076 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.076 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.077 DEBUG:boto:Method: GET 2024-03-04T16:33:57.077 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.077 DEBUG:boto:Data: 2024-03-04T16:33:57.077 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.077 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.077 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.077 DEBUG:boto:Params: {} 2024-03-04T16:33:57.077 DEBUG:boto:Token: None 2024-03-04T16:33:57.077 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8bcc7bb60ca6f90-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.080 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.081 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.081 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.081 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.081 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.081 DEBUG:boto:Method: GET 2024-03-04T16:33:57.081 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.081 DEBUG:boto:Data: 2024-03-04T16:33:57.082 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.082 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.082 DEBUG:boto:Params: {} 2024-03-04T16:33:57.082 DEBUG:boto:Token: None 2024-03-04T16:33:57.082 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.082 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000038b9ed978a8907-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.128 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.128 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.129 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.129 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.129 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.129 DEBUG:boto:Data: 2024-03-04T16:33:57.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.129 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.129 DEBUG:boto:Params: {} 2024-03-04T16:33:57.129 DEBUG:boto:Token: None 2024-03-04T16:33:57.130 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003788ba1304be3ec0-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.134 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.134 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.134 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.134 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.134 DEBUG:boto:Data: 2024-03-04T16:33:57.134 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.134 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.134 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.134 DEBUG:boto:Params: {} 2024-03-04T16:33:57.135 DEBUG:boto:Token: None 2024-03-04T16:33:57.135 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d74565133ccb9f70-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.138 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.138 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.138 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.139 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.139 DEBUG:boto:Method: GET 2024-03-04T16:33:57.139 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.139 DEBUG:boto:Data: 2024-03-04T16:33:57.139 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.139 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.139 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.139 DEBUG:boto:Params: {} 2024-03-04T16:33:57.139 DEBUG:boto:Token: None 2024-03-04T16:33:57.139 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.140 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f46fb88fd3c0d67d-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.188 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.188 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.188 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.188 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.189 DEBUG:boto:Method: GET 2024-03-04T16:33:57.189 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.189 DEBUG:boto:Data: 2024-03-04T16:33:57.189 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.189 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.189 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.189 DEBUG:boto:Params: {} 2024-03-04T16:33:57.189 DEBUG:boto:Token: None 2024-03-04T16:33:57.189 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.189 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000017fc4da50aab833-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.192 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.192 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.192 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.192 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.192 DEBUG:boto:Method: GET 2024-03-04T16:33:57.192 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.193 DEBUG:boto:Data: 2024-03-04T16:33:57.193 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.193 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.193 DEBUG:boto:Params: {} 2024-03-04T16:33:57.193 DEBUG:boto:Token: None 2024-03-04T16:33:57.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.193 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fefc62be5b7fde40-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.240 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.241 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.241 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.241 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.241 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.241 DEBUG:boto:Method: GET 2024-03-04T16:33:57.241 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.241 DEBUG:boto:Data: 2024-03-04T16:33:57.241 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.241 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.242 DEBUG:boto:Params: {} 2024-03-04T16:33:57.242 DEBUG:boto:Token: None 2024-03-04T16:33:57.242 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077444a316bb68963-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.245 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.245 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.246 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.246 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.246 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.246 DEBUG:boto:Method: GET 2024-03-04T16:33:57.246 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.246 DEBUG:boto:Data: 2024-03-04T16:33:57.246 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.246 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.247 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.247 DEBUG:boto:Params: {} 2024-03-04T16:33:57.247 DEBUG:boto:Token: None 2024-03-04T16:33:57.247 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.247 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ= 2024-03-04T16:33:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ=', 'Content-Length': '0'} 2024-03-04T16:33:57.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d859766ad1786a56-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.296 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.296 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.296 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.296 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.297 DEBUG:boto:Method: GET 2024-03-04T16:33:57.297 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.297 DEBUG:boto:Data: 2024-03-04T16:33:57.297 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.297 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.297 DEBUG:boto:Params: {} 2024-03-04T16:33:57.297 DEBUG:boto:Token: None 2024-03-04T16:33:57.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ= 2024-03-04T16:33:57.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ=', 'Content-Length': '0'} 2024-03-04T16:33:57.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c1b58503e1a2e41-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.300 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.300 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.300 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.300 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.300 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.300 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.300 DEBUG:boto:Data: 2024-03-04T16:33:57.301 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.301 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.301 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.301 DEBUG:boto:Params: {} 2024-03-04T16:33:57.301 DEBUG:boto:Token: None 2024-03-04T16:33:57.301 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.301 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dx6Pq/3zA5y8IQ39cKgIJUxP1Iw= 2024-03-04T16:33:57.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dx6Pq/3zA5y8IQ39cKgIJUxP1Iw=', 'Content-Length': '0'} 2024-03-04T16:33:57.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000767b5a49deddb92a-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.303 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.303 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.303 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.304 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.304 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.304 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.304 DEBUG:boto:Data: 2024-03-04T16:33:57.304 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.304 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.304 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.304 DEBUG:boto:Params: {} 2024-03-04T16:33:57.304 DEBUG:boto:Token: None 2024-03-04T16:33:57.304 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.305 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dx6Pq/3zA5y8IQ39cKgIJUxP1Iw= 2024-03-04T16:33:57.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dx6Pq/3zA5y8IQ39cKgIJUxP1Iw=', 'Content-Length': '0'} 2024-03-04T16:33:57.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e734808232f36ade-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.307 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.307 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.307 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.307 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.307 DEBUG:boto:Method: GET 2024-03-04T16:33:57.307 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.307 DEBUG:boto:Data: 2024-03-04T16:33:57.307 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.307 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.307 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.307 DEBUG:boto:Params: {} 2024-03-04T16:33:57.308 DEBUG:boto:Token: None 2024-03-04T16:33:57.308 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.308 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ= 2024-03-04T16:33:57.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ=', 'Content-Length': '0'} 2024-03-04T16:33:57.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000260e3e73ca13d237-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.356 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.356 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.356 DEBUG:boto:path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.356 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.356 DEBUG:boto:Method: GET 2024-03-04T16:33:57.357 DEBUG:boto:Path: /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.357 DEBUG:boto:Data: 2024-03-04T16:33:57.357 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.357 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.357 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.357 DEBUG:boto:Params: {} 2024-03-04T16:33:57.357 DEBUG:boto:Token: None 2024-03-04T16:33:57.357 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH 2024-03-04T16:33:57.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ= 2024-03-04T16:33:57.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:15/q3ExNBJrKEw8QXdprSzDufAQ=', 'Content-Length': '0'} 2024-03-04T16:33:57.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'RP2aX6PN4tqCxXe-c6sZV5.fhRwjvzH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb6771f21b39db9c-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.360 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.360 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.360 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.360 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.360 DEBUG:boto:Method: GET 2024-03-04T16:33:57.360 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.360 DEBUG:boto:Data: 2024-03-04T16:33:57.360 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.360 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.361 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.361 DEBUG:boto:Params: {} 2024-03-04T16:33:57.361 DEBUG:boto:Token: None 2024-03-04T16:33:57.361 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.361 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002820e40a6e33f6fe-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.404 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.404 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.405 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.405 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.405 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.405 DEBUG:boto:Method: GET 2024-03-04T16:33:57.405 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.405 DEBUG:boto:Data: 2024-03-04T16:33:57.405 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.405 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.405 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.405 DEBUG:boto:Params: {} 2024-03-04T16:33:57.406 DEBUG:boto:Token: None 2024-03-04T16:33:57.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004df300e061b278f2-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.408 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.409 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.409 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.409 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.409 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.409 DEBUG:boto:Data: 2024-03-04T16:33:57.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.409 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.410 DEBUG:boto:Params: {} 2024-03-04T16:33:57.410 DEBUG:boto:Token: None 2024-03-04T16:33:57.410 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4382132513a3cbd-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.414 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.414 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.414 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.414 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.414 DEBUG:boto:Data: 2024-03-04T16:33:57.414 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.414 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.414 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.414 DEBUG:boto:Params: {} 2024-03-04T16:33:57.414 DEBUG:boto:Token: None 2024-03-04T16:33:57.415 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000968308f58f789d64-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.418 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.418 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.418 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.418 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.419 DEBUG:boto:Method: GET 2024-03-04T16:33:57.419 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.419 DEBUG:boto:Data: 2024-03-04T16:33:57.419 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.419 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.419 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.419 DEBUG:boto:Params: {} 2024-03-04T16:33:57.419 DEBUG:boto:Token: None 2024-03-04T16:33:57.419 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009902d0deabb4e5f4-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.468 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.468 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.468 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.468 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.469 DEBUG:boto:Method: GET 2024-03-04T16:33:57.469 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.469 DEBUG:boto:Data: 2024-03-04T16:33:57.469 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.469 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.469 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.469 DEBUG:boto:Params: {} 2024-03-04T16:33:57.470 DEBUG:boto:Token: None 2024-03-04T16:33:57.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037ead01dd2603f53-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.472 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.472 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.472 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.472 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.473 DEBUG:boto:Method: GET 2024-03-04T16:33:57.473 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.473 DEBUG:boto:Data: 2024-03-04T16:33:57.473 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.473 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.473 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.473 DEBUG:boto:Params: {} 2024-03-04T16:33:57.473 DEBUG:boto:Token: None 2024-03-04T16:33:57.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08733ffb6b3889d-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.524 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.525 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.525 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.525 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.525 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.525 DEBUG:boto:Method: GET 2024-03-04T16:33:57.525 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.525 DEBUG:boto:Data: 2024-03-04T16:33:57.525 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.525 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.526 DEBUG:boto:Params: {} 2024-03-04T16:33:57.526 DEBUG:boto:Token: None 2024-03-04T16:33:57.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9b7c323f802e87-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.529 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.530 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.530 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.530 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.530 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.530 DEBUG:boto:Method: GET 2024-03-04T16:33:57.530 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.530 DEBUG:boto:Data: 2024-03-04T16:33:57.530 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.530 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.530 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.531 DEBUG:boto:Params: {} 2024-03-04T16:33:57.531 DEBUG:boto:Token: None 2024-03-04T16:33:57.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI= 2024-03-04T16:33:57.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI=', 'Content-Length': '0'} 2024-03-04T16:33:57.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003893e8718ab402a3-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.580 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.580 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.580 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.580 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.580 DEBUG:boto:Method: GET 2024-03-04T16:33:57.581 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.581 DEBUG:boto:Data: 2024-03-04T16:33:57.581 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.581 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.581 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.581 DEBUG:boto:Params: {} 2024-03-04T16:33:57.581 DEBUG:boto:Token: None 2024-03-04T16:33:57.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI= 2024-03-04T16:33:57.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI=', 'Content-Length': '0'} 2024-03-04T16:33:57.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e22619cade5aa12-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.624 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.624 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.624 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.625 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.625 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.625 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.625 DEBUG:boto:Data: 2024-03-04T16:33:57.625 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.625 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.625 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.625 DEBUG:boto:Params: {} 2024-03-04T16:33:57.625 DEBUG:boto:Token: None 2024-03-04T16:33:57.625 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DuAx2HqMQs0IS6b8in3znshF94w= 2024-03-04T16:33:57.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DuAx2HqMQs0IS6b8in3znshF94w=', 'Content-Length': '0'} 2024-03-04T16:33:57.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7ebb687488ce383-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.628 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.628 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.628 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.628 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.628 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.628 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.628 DEBUG:boto:Data: 2024-03-04T16:33:57.628 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.629 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.629 DEBUG:boto:Params: {} 2024-03-04T16:33:57.629 DEBUG:boto:Token: None 2024-03-04T16:33:57.629 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.629 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DuAx2HqMQs0IS6b8in3znshF94w= 2024-03-04T16:33:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DuAx2HqMQs0IS6b8in3znshF94w=', 'Content-Length': '0'} 2024-03-04T16:33:57.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4be4d3be08b720d-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.632 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.632 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.632 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.632 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.632 DEBUG:boto:Method: GET 2024-03-04T16:33:57.632 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.632 DEBUG:boto:Data: 2024-03-04T16:33:57.632 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.632 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.632 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.633 DEBUG:boto:Params: {} 2024-03-04T16:33:57.633 DEBUG:boto:Token: None 2024-03-04T16:33:57.633 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI= 2024-03-04T16:33:57.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI=', 'Content-Length': '0'} 2024-03-04T16:33:57.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3a8ff2d49707f11-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.676 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.676 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.676 DEBUG:boto:path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.676 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.677 DEBUG:boto:Method: GET 2024-03-04T16:33:57.677 DEBUG:boto:Path: /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.677 DEBUG:boto:Data: 2024-03-04T16:33:57.677 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.677 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.677 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.677 DEBUG:boto:Params: {} 2024-03-04T16:33:57.677 DEBUG:boto:Token: None 2024-03-04T16:33:57.677 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=WqFQw08WT4szpVVRJMZzxcn8IIQaJrX 2024-03-04T16:33:57.678 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI= 2024-03-04T16:33:57.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2HnRlQgkuzB+AQkNgO10DR/+naI=', 'Content-Length': '0'} 2024-03-04T16:33:57.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'WqFQw08WT4szpVVRJMZzxcn8IIQaJrX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047e5e0d7ea82d6c8-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.728 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.728 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.728 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.729 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.729 DEBUG:boto:Method: GET 2024-03-04T16:33:57.729 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.729 DEBUG:boto:Data: 2024-03-04T16:33:57.729 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.729 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.729 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.729 DEBUG:boto:Params: {} 2024-03-04T16:33:57.729 DEBUG:boto:Token: None 2024-03-04T16:33:57.729 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df63163d47095be7-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.780 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.780 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.781 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.781 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.781 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.781 DEBUG:boto:Method: GET 2024-03-04T16:33:57.781 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.781 DEBUG:boto:Data: 2024-03-04T16:33:57.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.781 DEBUG:boto:Params: {} 2024-03-04T16:33:57.781 DEBUG:boto:Token: None 2024-03-04T16:33:57.782 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b08e0aa7bc7f5b-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.785 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.785 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.785 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.785 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.785 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.785 DEBUG:boto:Data: 2024-03-04T16:33:57.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.786 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.786 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.786 DEBUG:boto:Params: {} 2024-03-04T16:33:57.786 DEBUG:boto:Token: None 2024-03-04T16:33:57.786 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.786 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044ea9a4778c47c74-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.790 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.790 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.790 DEBUG:boto:Method: HEAD 2024-03-04T16:33:57.790 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:57.790 DEBUG:boto:Data: 2024-03-04T16:33:57.790 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.790 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.790 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.790 DEBUG:boto:Params: {} 2024-03-04T16:33:57.790 DEBUG:boto:Token: None 2024-03-04T16:33:57.790 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj 2024-03-04T16:33:57.791 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs= 2024-03-04T16:33:57.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GUZsz60T20KNE0CI/ZsJUygeyCs=', 'Content-Length': '0'} 2024-03-04T16:33:57.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000252259044f7b708a-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.794 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.794 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.794 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.795 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.795 DEBUG:boto:Method: GET 2024-03-04T16:33:57.795 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.795 DEBUG:boto:Data: 2024-03-04T16:33:57.795 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.795 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.795 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.795 DEBUG:boto:Params: {} 2024-03-04T16:33:57.795 DEBUG:boto:Token: None 2024-03-04T16:33:57.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a9d9d7e59279f04-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.840 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.840 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.840 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.840 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.841 DEBUG:boto:Method: GET 2024-03-04T16:33:57.841 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.841 DEBUG:boto:Data: 2024-03-04T16:33:57.841 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.841 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.841 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.841 DEBUG:boto:Params: {} 2024-03-04T16:33:57.841 DEBUG:boto:Token: None 2024-03-04T16:33:57.841 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:57.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk= 2024-03-04T16:33:57.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L2EyWjk879z0lghEorRfKWlBAHk=', 'Content-Length': '0'} 2024-03-04T16:33:57.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000897ad9719aca47de-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.884 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.884 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.884 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.884 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.885 DEBUG:boto:Method: GET 2024-03-04T16:33:57.885 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.885 DEBUG:boto:Data: 2024-03-04T16:33:57.885 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.885 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.885 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.885 DEBUG:boto:Params: {} 2024-03-04T16:33:57.885 DEBUG:boto:Token: None 2024-03-04T16:33:57.886 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701c9ed78d8fec08-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.932 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.932 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:57.933 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:57.933 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.933 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:57.933 DEBUG:boto:Method: GET 2024-03-04T16:33:57.933 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:57.933 DEBUG:boto:Data: 2024-03-04T16:33:57.933 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.933 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.933 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.933 DEBUG:boto:Params: {} 2024-03-04T16:33:57.934 DEBUG:boto:Token: None 2024-03-04T16:33:57.934 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/ 2024-03-04T16:33:57.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE= 2024-03-04T16:33:57.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FUsd+VSpr4qyuw9IxiCWHsEjuJE=', 'Content-Length': '0'} 2024-03-04T16:33:57.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7b083230c4a226-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.937 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:57.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:57.937 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.938 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.938 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.938 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.938 DEBUG:boto:Method: GET 2024-03-04T16:33:57.938 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.938 DEBUG:boto:Data: 2024-03-04T16:33:57.938 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.938 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:57.938 DEBUG:boto:Port: 8001 2024-03-04T16:33:57.938 DEBUG:boto:Params: {} 2024-03-04T16:33:57.938 DEBUG:boto:Token: None 2024-03-04T16:33:57.939 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.939 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rCk0Mrpaf4BFPhaZt6WoWe/lH98= 2024-03-04T16:33:57.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rCk0Mrpaf4BFPhaZt6WoWe/lH98=', 'Content-Length': '0'} 2024-03-04T16:33:57.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5a636416fd2f121-0065e5f7f5-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:57.984 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:57.984 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:57.984 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.985 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.985 DEBUG:boto:Method: GET 2024-03-04T16:33:57.985 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.985 DEBUG:boto:Data: 2024-03-04T16:33:57.985 DEBUG:boto:Headers: {} 2024-03-04T16:33:57.985 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:57.985 DEBUG:boto:Port: 8000 2024-03-04T16:33:57.985 DEBUG:boto:Params: {} 2024-03-04T16:33:57.985 DEBUG:boto:Token: None 2024-03-04T16:33:57.985 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:57 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:57.986 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rCk0Mrpaf4BFPhaZt6WoWe/lH98= 2024-03-04T16:33:57.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rCk0Mrpaf4BFPhaZt6WoWe/lH98=', 'Content-Length': '0'} 2024-03-04T16:33:57.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002a8250e4f91246b-0065e5f7f5-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.028 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.028 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.028 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.028 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.029 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.029 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.029 DEBUG:boto:Data: 2024-03-04T16:33:58.029 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.029 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.029 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.029 DEBUG:boto:Params: {} 2024-03-04T16:33:58.029 DEBUG:boto:Token: None 2024-03-04T16:33:58.029 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.029 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZapWMdSAdtOrl3kMTKqq8/iTtgM= 2024-03-04T16:33:58.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZapWMdSAdtOrl3kMTKqq8/iTtgM=', 'Content-Length': '0'} 2024-03-04T16:33:58.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039cd64f7500a94a1-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.032 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.032 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.032 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.032 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.032 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.032 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.032 DEBUG:boto:Data: 2024-03-04T16:33:58.032 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.033 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.033 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.033 DEBUG:boto:Params: {} 2024-03-04T16:33:58.033 DEBUG:boto:Token: None 2024-03-04T16:33:58.033 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.033 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZapWMdSAdtOrl3kMTKqq8/iTtgM= 2024-03-04T16:33:58.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZapWMdSAdtOrl3kMTKqq8/iTtgM=', 'Content-Length': '0'} 2024-03-04T16:33:58.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036dd865f37cc71ae-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.035 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.035 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.036 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.036 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.036 DEBUG:boto:Method: GET 2024-03-04T16:33:58.036 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.036 DEBUG:boto:Data: 2024-03-04T16:33:58.036 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.036 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.036 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.036 DEBUG:boto:Params: {} 2024-03-04T16:33:58.036 DEBUG:boto:Token: None 2024-03-04T16:33:58.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.037 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eIgjUwrDRfDPDVaiIAuPLb1YGIk= 2024-03-04T16:33:58.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eIgjUwrDRfDPDVaiIAuPLb1YGIk=', 'Content-Length': '0'} 2024-03-04T16:33:58.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc566d6d4afdca97-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.080 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.081 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.081 DEBUG:boto:path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.081 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.081 DEBUG:boto:Method: GET 2024-03-04T16:33:58.081 DEBUG:boto:Path: /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.082 DEBUG:boto:Data: 2024-03-04T16:33:58.082 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.082 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.082 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.082 DEBUG:boto:Params: {} 2024-03-04T16:33:58.082 DEBUG:boto:Token: None 2024-03-04T16:33:58.083 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=60c45DIWYEmYXg2q5fpDBATrHDwl4Gp 2024-03-04T16:33:58.083 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:eIgjUwrDRfDPDVaiIAuPLb1YGIk= 2024-03-04T16:33:58.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:eIgjUwrDRfDPDVaiIAuPLb1YGIk=', 'Content-Length': '0'} 2024-03-04T16:33:58.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '60c45DIWYEmYXg2q5fpDBATrHDwl4Gp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a77070a77028ffdb-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.128 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.129 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.129 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.129 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.129 DEBUG:boto:Method: GET 2024-03-04T16:33:58.129 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.129 DEBUG:boto:Data: 2024-03-04T16:33:58.129 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.129 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.129 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.129 DEBUG:boto:Params: {} 2024-03-04T16:33:58.130 DEBUG:boto:Token: None 2024-03-04T16:33:58.130 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.130 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c758d9c517221ee9-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.176 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.176 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.177 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.177 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.177 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.177 DEBUG:boto:Method: GET 2024-03-04T16:33:58.177 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.177 DEBUG:boto:Data: 2024-03-04T16:33:58.177 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.177 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.177 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.177 DEBUG:boto:Params: {} 2024-03-04T16:33:58.178 DEBUG:boto:Token: None 2024-03-04T16:33:58.178 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.178 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c79744a799deec22-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.181 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.182 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.182 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.182 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.182 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.182 DEBUG:boto:Data: 2024-03-04T16:33:58.182 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.182 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.182 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.183 DEBUG:boto:Params: {} 2024-03-04T16:33:58.183 DEBUG:boto:Token: None 2024-03-04T16:33:58.183 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.183 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000746cb7c449786bf7-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.187 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.187 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.187 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.187 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.187 DEBUG:boto:Data: 2024-03-04T16:33:58.187 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.187 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.187 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.187 DEBUG:boto:Params: {} 2024-03-04T16:33:58.188 DEBUG:boto:Token: None 2024-03-04T16:33:58.188 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.188 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000854ec0c5556adc39-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.192 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.192 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.192 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.192 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.192 DEBUG:boto:Method: GET 2024-03-04T16:33:58.193 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.193 DEBUG:boto:Data: 2024-03-04T16:33:58.193 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.193 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.193 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.193 DEBUG:boto:Params: {} 2024-03-04T16:33:58.193 DEBUG:boto:Token: None 2024-03-04T16:33:58.193 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.193 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b851adc11388c70a-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.236 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.236 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.236 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.237 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.237 DEBUG:boto:Method: GET 2024-03-04T16:33:58.237 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.237 DEBUG:boto:Data: 2024-03-04T16:33:58.237 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.237 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.237 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.237 DEBUG:boto:Params: {} 2024-03-04T16:33:58.237 DEBUG:boto:Token: None 2024-03-04T16:33:58.237 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.238 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b71cf7aac11ee172-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.240 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.240 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.240 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.240 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.240 DEBUG:boto:Method: GET 2024-03-04T16:33:58.240 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.240 DEBUG:boto:Data: 2024-03-04T16:33:58.240 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.240 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.241 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.241 DEBUG:boto:Params: {} 2024-03-04T16:33:58.241 DEBUG:boto:Token: None 2024-03-04T16:33:58.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000721905784909fb9c-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.284 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.285 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.285 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.285 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.285 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.285 DEBUG:boto:Method: GET 2024-03-04T16:33:58.285 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.285 DEBUG:boto:Data: 2024-03-04T16:33:58.285 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.285 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.285 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.286 DEBUG:boto:Params: {} 2024-03-04T16:33:58.286 DEBUG:boto:Token: None 2024-03-04T16:33:58.286 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.286 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41124b7b2c13905-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.289 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.289 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.290 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.290 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.290 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.290 DEBUG:boto:Method: GET 2024-03-04T16:33:58.290 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.290 DEBUG:boto:Data: 2024-03-04T16:33:58.290 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.290 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.290 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.290 DEBUG:boto:Params: {} 2024-03-04T16:33:58.290 DEBUG:boto:Token: None 2024-03-04T16:33:58.291 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.291 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o= 2024-03-04T16:33:58.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o=', 'Content-Length': '0'} 2024-03-04T16:33:58.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000159935ad24c162cf-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.336 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.336 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.337 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.337 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.337 DEBUG:boto:Method: GET 2024-03-04T16:33:58.337 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.337 DEBUG:boto:Data: 2024-03-04T16:33:58.337 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.337 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.337 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.337 DEBUG:boto:Params: {} 2024-03-04T16:33:58.337 DEBUG:boto:Token: None 2024-03-04T16:33:58.338 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.338 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o= 2024-03-04T16:33:58.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o=', 'Content-Length': '0'} 2024-03-04T16:33:58.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044db3d7017d15f99-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.340 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.340 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.340 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.340 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.341 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.341 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.341 DEBUG:boto:Data: 2024-03-04T16:33:58.341 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.341 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.341 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.341 DEBUG:boto:Params: {} 2024-03-04T16:33:58.341 DEBUG:boto:Token: None 2024-03-04T16:33:58.341 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.341 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9PAYoaTJFjtpfZCgka73v3NoJKk= 2024-03-04T16:33:58.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9PAYoaTJFjtpfZCgka73v3NoJKk=', 'Content-Length': '0'} 2024-03-04T16:33:58.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002335a54621153fa5-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.344 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.344 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.344 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.344 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.344 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.344 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.344 DEBUG:boto:Data: 2024-03-04T16:33:58.345 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.345 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.345 DEBUG:boto:Params: {} 2024-03-04T16:33:58.345 DEBUG:boto:Token: None 2024-03-04T16:33:58.345 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9PAYoaTJFjtpfZCgka73v3NoJKk= 2024-03-04T16:33:58.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9PAYoaTJFjtpfZCgka73v3NoJKk=', 'Content-Length': '0'} 2024-03-04T16:33:58.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fbbeccf768156bd-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.348 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.348 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.348 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.348 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.348 DEBUG:boto:Method: GET 2024-03-04T16:33:58.348 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.348 DEBUG:boto:Data: 2024-03-04T16:33:58.348 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.348 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.348 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.348 DEBUG:boto:Params: {} 2024-03-04T16:33:58.349 DEBUG:boto:Token: None 2024-03-04T16:33:58.349 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.349 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o= 2024-03-04T16:33:58.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o=', 'Content-Length': '0'} 2024-03-04T16:33:58.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085e6076e6a4e0654-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.396 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.396 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.396 DEBUG:boto:path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.396 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.397 DEBUG:boto:Method: GET 2024-03-04T16:33:58.397 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.397 DEBUG:boto:Data: 2024-03-04T16:33:58.397 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.397 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.397 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.397 DEBUG:boto:Params: {} 2024-03-04T16:33:58.397 DEBUG:boto:Token: None 2024-03-04T16:33:58.397 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Y9VfGbVuWMLRnERQj5tDOof79HPhWHl 2024-03-04T16:33:58.398 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o= 2024-03-04T16:33:58.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:741HCIfU4/WnEo6k6D/tIXYix8o=', 'Content-Length': '0'} 2024-03-04T16:33:58.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Y9VfGbVuWMLRnERQj5tDOof79HPhWHl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000832cfbadefcc577-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.400 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.400 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.400 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.400 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.401 DEBUG:boto:Method: GET 2024-03-04T16:33:58.401 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.401 DEBUG:boto:Data: 2024-03-04T16:33:58.401 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.401 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.401 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.401 DEBUG:boto:Params: {} 2024-03-04T16:33:58.401 DEBUG:boto:Token: None 2024-03-04T16:33:58.401 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0fdfd57a211c487-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.448 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.449 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.449 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.449 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.449 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.449 DEBUG:boto:Method: GET 2024-03-04T16:33:58.449 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.449 DEBUG:boto:Data: 2024-03-04T16:33:58.449 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.449 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.449 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.449 DEBUG:boto:Params: {} 2024-03-04T16:33:58.450 DEBUG:boto:Token: None 2024-03-04T16:33:58.450 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d36771db0b2d63a3-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.453 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.453 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.453 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.453 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.454 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.454 DEBUG:boto:Data: 2024-03-04T16:33:58.454 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.454 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.454 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.454 DEBUG:boto:Params: {} 2024-03-04T16:33:58.454 DEBUG:boto:Token: None 2024-03-04T16:33:58.454 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdae0a0f6378ad53-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.458 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.458 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.458 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.458 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.458 DEBUG:boto:Data: 2024-03-04T16:33:58.458 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.458 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.459 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.459 DEBUG:boto:Params: {} 2024-03-04T16:33:58.459 DEBUG:boto:Token: None 2024-03-04T16:33:58.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b027c4a3c5218549-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.463 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.463 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.463 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.463 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.463 DEBUG:boto:Method: GET 2024-03-04T16:33:58.463 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.463 DEBUG:boto:Data: 2024-03-04T16:33:58.463 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.464 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.464 DEBUG:boto:Params: {} 2024-03-04T16:33:58.464 DEBUG:boto:Token: None 2024-03-04T16:33:58.464 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.464 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099eac3b620055fad-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.508 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.508 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.508 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.508 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.509 DEBUG:boto:Method: GET 2024-03-04T16:33:58.509 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.509 DEBUG:boto:Data: 2024-03-04T16:33:58.509 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.509 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.509 DEBUG:boto:Params: {} 2024-03-04T16:33:58.509 DEBUG:boto:Token: None 2024-03-04T16:33:58.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce4de9b68121565b-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.512 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.512 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.512 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.512 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.512 DEBUG:boto:Method: GET 2024-03-04T16:33:58.512 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.512 DEBUG:boto:Data: 2024-03-04T16:33:58.512 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.513 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.513 DEBUG:boto:Params: {} 2024-03-04T16:33:58.513 DEBUG:boto:Token: None 2024-03-04T16:33:58.513 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa09bb2f41b557c-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.560 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.560 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.561 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.561 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.561 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.561 DEBUG:boto:Method: GET 2024-03-04T16:33:58.561 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.561 DEBUG:boto:Data: 2024-03-04T16:33:58.561 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.561 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.561 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.561 DEBUG:boto:Params: {} 2024-03-04T16:33:58.562 DEBUG:boto:Token: None 2024-03-04T16:33:58.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004564f20eb70c6fd-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.565 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.565 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.565 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.565 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.566 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.566 DEBUG:boto:Method: GET 2024-03-04T16:33:58.566 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.566 DEBUG:boto:Data: 2024-03-04T16:33:58.566 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.566 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.566 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.566 DEBUG:boto:Params: {} 2024-03-04T16:33:58.566 DEBUG:boto:Token: None 2024-03-04T16:33:58.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco= 2024-03-04T16:33:58.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco=', 'Content-Length': '0'} 2024-03-04T16:33:58.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003346437a46d7e8cd-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.569 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.569 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.569 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.569 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.569 DEBUG:boto:Method: GET 2024-03-04T16:33:58.569 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.569 DEBUG:boto:Data: 2024-03-04T16:33:58.569 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.569 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.569 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.570 DEBUG:boto:Params: {} 2024-03-04T16:33:58.570 DEBUG:boto:Token: None 2024-03-04T16:33:58.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco= 2024-03-04T16:33:58.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco=', 'Content-Length': '0'} 2024-03-04T16:33:58.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e07591ba0b920623-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.612 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.612 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.613 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.613 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.613 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.613 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.613 DEBUG:boto:Data: 2024-03-04T16:33:58.613 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.613 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.614 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.614 DEBUG:boto:Params: {} 2024-03-04T16:33:58.614 DEBUG:boto:Token: None 2024-03-04T16:33:58.614 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sXBISUZChC2fKKz9Wv71zcm7w9w= 2024-03-04T16:33:58.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sXBISUZChC2fKKz9Wv71zcm7w9w=', 'Content-Length': '0'} 2024-03-04T16:33:58.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000039d1500ed2798ff-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.617 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.617 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.617 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.617 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.618 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.618 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.618 DEBUG:boto:Data: 2024-03-04T16:33:58.618 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.618 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.618 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.618 DEBUG:boto:Params: {} 2024-03-04T16:33:58.618 DEBUG:boto:Token: None 2024-03-04T16:33:58.618 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.619 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sXBISUZChC2fKKz9Wv71zcm7w9w= 2024-03-04T16:33:58.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sXBISUZChC2fKKz9Wv71zcm7w9w=', 'Content-Length': '0'} 2024-03-04T16:33:58.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091fc32d8c0ee1f03-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.621 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.621 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.621 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.621 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.621 DEBUG:boto:Method: GET 2024-03-04T16:33:58.621 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.621 DEBUG:boto:Data: 2024-03-04T16:33:58.622 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.622 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.622 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.622 DEBUG:boto:Params: {} 2024-03-04T16:33:58.622 DEBUG:boto:Token: None 2024-03-04T16:33:58.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco= 2024-03-04T16:33:58.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco=', 'Content-Length': '0'} 2024-03-04T16:33:58.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b59c6855889cb0b0-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.625 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.625 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.625 DEBUG:boto:path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.625 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.625 DEBUG:boto:Method: GET 2024-03-04T16:33:58.625 DEBUG:boto:Path: /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.625 DEBUG:boto:Data: 2024-03-04T16:33:58.625 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.625 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.626 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.626 DEBUG:boto:Params: {} 2024-03-04T16:33:58.626 DEBUG:boto:Token: None 2024-03-04T16:33:58.626 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN 2024-03-04T16:33:58.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco= 2024-03-04T16:33:58.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6motM11PrTJBy0XHuJ+C1TucEco=', 'Content-Length': '0'} 2024-03-04T16:33:58.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'us1Q-hNLD9lb1a0qvK1BMsAcZBvQaaN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000765f6b468b7bd78a-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.672 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.672 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.672 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.672 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.673 DEBUG:boto:Method: GET 2024-03-04T16:33:58.673 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.673 DEBUG:boto:Data: 2024-03-04T16:33:58.673 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.673 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.673 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.673 DEBUG:boto:Params: {} 2024-03-04T16:33:58.673 DEBUG:boto:Token: None 2024-03-04T16:33:58.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.674 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d980c2d2c2edb995-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.677 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.677 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.677 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.678 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.678 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.678 DEBUG:boto:Method: GET 2024-03-04T16:33:58.678 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.678 DEBUG:boto:Data: 2024-03-04T16:33:58.678 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.679 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.679 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.679 DEBUG:boto:Params: {} 2024-03-04T16:33:58.679 DEBUG:boto:Token: None 2024-03-04T16:33:58.680 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.680 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ceb9d100d7c4e97-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.724 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.725 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.725 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.725 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.725 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.725 DEBUG:boto:Data: 2024-03-04T16:33:58.725 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.725 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.725 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.725 DEBUG:boto:Params: {} 2024-03-04T16:33:58.725 DEBUG:boto:Token: None 2024-03-04T16:33:58.726 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f5a6dc93651e87f-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.729 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.730 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.730 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.730 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.730 DEBUG:boto:Data: 2024-03-04T16:33:58.730 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.730 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.730 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.730 DEBUG:boto:Params: {} 2024-03-04T16:33:58.730 DEBUG:boto:Token: None 2024-03-04T16:33:58.730 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.730 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c4067b81fb1c376-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.734 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.735 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.735 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.735 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.735 DEBUG:boto:Method: GET 2024-03-04T16:33:58.735 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.735 DEBUG:boto:Data: 2024-03-04T16:33:58.735 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.735 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.735 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.735 DEBUG:boto:Params: {} 2024-03-04T16:33:58.735 DEBUG:boto:Token: None 2024-03-04T16:33:58.736 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.736 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b699eedb8dc7b09-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.780 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.780 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.780 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.781 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.781 DEBUG:boto:Method: GET 2024-03-04T16:33:58.781 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.781 DEBUG:boto:Data: 2024-03-04T16:33:58.781 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.781 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.781 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.781 DEBUG:boto:Params: {} 2024-03-04T16:33:58.781 DEBUG:boto:Token: None 2024-03-04T16:33:58.781 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.782 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000097deadbad948155-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.784 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.784 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.784 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.784 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.784 DEBUG:boto:Method: GET 2024-03-04T16:33:58.785 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.785 DEBUG:boto:Data: 2024-03-04T16:33:58.785 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.785 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.785 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.785 DEBUG:boto:Params: {} 2024-03-04T16:33:58.785 DEBUG:boto:Token: None 2024-03-04T16:33:58.785 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682fc6c4108773c4-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.789 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.789 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.789 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.789 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.789 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.789 DEBUG:boto:Method: GET 2024-03-04T16:33:58.790 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.790 DEBUG:boto:Data: 2024-03-04T16:33:58.790 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.790 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.790 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.790 DEBUG:boto:Params: {} 2024-03-04T16:33:58.790 DEBUG:boto:Token: None 2024-03-04T16:33:58.790 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.790 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d2546a63fee61a5-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.793 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.794 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.794 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.794 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.794 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.794 DEBUG:boto:Method: GET 2024-03-04T16:33:58.794 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.794 DEBUG:boto:Data: 2024-03-04T16:33:58.794 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.794 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.795 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.795 DEBUG:boto:Params: {} 2024-03-04T16:33:58.795 DEBUG:boto:Token: None 2024-03-04T16:33:58.795 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.795 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68= 2024-03-04T16:33:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68=', 'Content-Length': '0'} 2024-03-04T16:33:58.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6dc54871af44cc7-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.844 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.844 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.844 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.844 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.845 DEBUG:boto:Method: GET 2024-03-04T16:33:58.845 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.845 DEBUG:boto:Data: 2024-03-04T16:33:58.845 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.845 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.845 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.845 DEBUG:boto:Params: {} 2024-03-04T16:33:58.845 DEBUG:boto:Token: None 2024-03-04T16:33:58.845 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68= 2024-03-04T16:33:58.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68=', 'Content-Length': '0'} 2024-03-04T16:33:58.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f07e699534958ce3-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.848 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.848 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.848 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.848 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.848 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.849 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.849 DEBUG:boto:Data: 2024-03-04T16:33:58.849 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.849 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.849 DEBUG:boto:Params: {} 2024-03-04T16:33:58.849 DEBUG:boto:Token: None 2024-03-04T16:33:58.849 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Cl/pJL16vEZU+e4dKce7O3QF3U= 2024-03-04T16:33:58.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Cl/pJL16vEZU+e4dKce7O3QF3U=', 'Content-Length': '0'} 2024-03-04T16:33:58.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000247649e9920482b1-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.851 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.852 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.852 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.852 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.852 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.852 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.852 DEBUG:boto:Data: 2024-03-04T16:33:58.852 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.852 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.852 DEBUG:boto:Params: {} 2024-03-04T16:33:58.853 DEBUG:boto:Token: None 2024-03-04T16:33:58.853 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1Cl/pJL16vEZU+e4dKce7O3QF3U= 2024-03-04T16:33:58.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1Cl/pJL16vEZU+e4dKce7O3QF3U=', 'Content-Length': '0'} 2024-03-04T16:33:58.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000111097bbec71637f-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.855 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.855 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.856 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.856 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.856 DEBUG:boto:Method: GET 2024-03-04T16:33:58.856 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.856 DEBUG:boto:Data: 2024-03-04T16:33:58.856 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.856 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.856 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.856 DEBUG:boto:Params: {} 2024-03-04T16:33:58.856 DEBUG:boto:Token: None 2024-03-04T16:33:58.856 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.857 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68= 2024-03-04T16:33:58.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68=', 'Content-Length': '0'} 2024-03-04T16:33:58.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000993ff9a98942503b-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.900 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.900 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.900 DEBUG:boto:path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.900 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.901 DEBUG:boto:Method: GET 2024-03-04T16:33:58.901 DEBUG:boto:Path: /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.901 DEBUG:boto:Data: 2024-03-04T16:33:58.901 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.901 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.901 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.901 DEBUG:boto:Params: {} 2024-03-04T16:33:58.901 DEBUG:boto:Token: None 2024-03-04T16:33:58.901 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=1slxKjMQBTNKx8wPzjpbP7zo48qatQb 2024-03-04T16:33:58.902 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68= 2024-03-04T16:33:58.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvlNOyYLojc2UnMRwoa1UPlqP68=', 'Content-Length': '0'} 2024-03-04T16:33:58.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', '1slxKjMQBTNKx8wPzjpbP7zo48qatQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8e2b0b9b1f9b2b9-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.904 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.904 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.904 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.904 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.904 DEBUG:boto:Method: GET 2024-03-04T16:33:58.904 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.904 DEBUG:boto:Data: 2024-03-04T16:33:58.905 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.905 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.905 DEBUG:boto:Params: {} 2024-03-04T16:33:58.905 DEBUG:boto:Token: None 2024-03-04T16:33:58.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.905 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054e126623fbadda2-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.909 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.909 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.909 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.909 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.909 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.909 DEBUG:boto:Method: GET 2024-03-04T16:33:58.909 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.909 DEBUG:boto:Data: 2024-03-04T16:33:58.910 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.910 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.910 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.910 DEBUG:boto:Params: {} 2024-03-04T16:33:58.910 DEBUG:boto:Token: None 2024-03-04T16:33:58.910 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.910 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030eb9b7fb4ade6d4-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.913 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.914 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.914 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.914 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.914 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.914 DEBUG:boto:Data: 2024-03-04T16:33:58.914 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.914 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.914 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.914 DEBUG:boto:Params: {} 2024-03-04T16:33:58.915 DEBUG:boto:Token: None 2024-03-04T16:33:58.915 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.915 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082de2a6a03eed26b-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.918 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.918 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.919 DEBUG:boto:Method: HEAD 2024-03-04T16:33:58.919 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:58.919 DEBUG:boto:Data: 2024-03-04T16:33:58.919 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.919 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.919 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.919 DEBUG:boto:Params: {} 2024-03-04T16:33:58.919 DEBUG:boto:Token: None 2024-03-04T16:33:58.919 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj 2024-03-04T16:33:58.919 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE= 2024-03-04T16:33:58.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vtb1UyXvUZYvaS4N3mbegzww6AE=', 'Content-Length': '0'} 2024-03-04T16:33:58.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000bbd3e4d3b96ff3-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.923 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.923 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.923 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.923 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.923 DEBUG:boto:Method: GET 2024-03-04T16:33:58.923 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.924 DEBUG:boto:Data: 2024-03-04T16:33:58.924 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.924 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.924 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.924 DEBUG:boto:Params: {} 2024-03-04T16:33:58.924 DEBUG:boto:Token: None 2024-03-04T16:33:58.924 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.924 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df0e5f1654d3754a-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.968 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.968 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.968 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.968 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.969 DEBUG:boto:Method: GET 2024-03-04T16:33:58.969 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.969 DEBUG:boto:Data: 2024-03-04T16:33:58.969 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.969 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.969 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.969 DEBUG:boto:Params: {} 2024-03-04T16:33:58.969 DEBUG:boto:Token: None 2024-03-04T16:33:58.969 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:58.969 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE= 2024-03-04T16:33:58.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YxsPKDRWEH3s1DfPu3uXgo/RqvE=', 'Content-Length': '0'} 2024-03-04T16:33:58.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000672d8ac1a12cd282-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.972 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.972 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.972 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.972 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.972 DEBUG:boto:Method: GET 2024-03-04T16:33:58.972 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.972 DEBUG:boto:Data: 2024-03-04T16:33:58.972 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.972 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.973 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.973 DEBUG:boto:Params: {} 2024-03-04T16:33:58.973 DEBUG:boto:Token: None 2024-03-04T16:33:58.973 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.973 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050deaf01d3edf620-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.976 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.979 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:58.979 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:58.979 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.980 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:58.980 DEBUG:boto:Method: GET 2024-03-04T16:33:58.980 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:58.980 DEBUG:boto:Data: 2024-03-04T16:33:58.980 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.980 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:58.980 DEBUG:boto:Port: 8000 2024-03-04T16:33:58.980 DEBUG:boto:Params: {} 2024-03-04T16:33:58.980 DEBUG:boto:Token: None 2024-03-04T16:33:58.981 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/ 2024-03-04T16:33:58.981 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc= 2024-03-04T16:33:58.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:To1N+AKLeyM7aS5ejrfVEaFYIKc=', 'Content-Length': '0'} 2024-03-04T16:33:58.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd97d6c400e923f-0065e5f7f6-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:58.984 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:58.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:58.984 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:58.984 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:58.984 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:58.984 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:58.985 DEBUG:boto:Method: GET 2024-03-04T16:33:58.985 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:58.985 DEBUG:boto:Data: 2024-03-04T16:33:58.985 DEBUG:boto:Headers: {} 2024-03-04T16:33:58.985 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:58.985 DEBUG:boto:Port: 8001 2024-03-04T16:33:58.985 DEBUG:boto:Params: {} 2024-03-04T16:33:58.985 DEBUG:boto:Token: None 2024-03-04T16:33:58.986 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:58 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:58.986 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xxnIYMfSVFF+APvU7D7WzlN5gI4= 2024-03-04T16:33:58.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:58 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xxnIYMfSVFF+APvU7D7WzlN5gI4=', 'Content-Length': '0'} 2024-03-04T16:33:58.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3be9514bb6440c1-0065e5f7f6-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.028 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.028 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.028 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.029 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.029 DEBUG:boto:Method: GET 2024-03-04T16:33:59.029 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.029 DEBUG:boto:Data: 2024-03-04T16:33:59.029 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.029 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.029 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.029 DEBUG:boto:Params: {} 2024-03-04T16:33:59.029 DEBUG:boto:Token: None 2024-03-04T16:33:59.029 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.030 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4= 2024-03-04T16:33:59.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4=', 'Content-Length': '0'} 2024-03-04T16:33:59.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e25bacd05815404-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.072 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.072 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.072 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.072 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.073 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.073 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.073 DEBUG:boto:Data: 2024-03-04T16:33:59.073 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.073 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.073 DEBUG:boto:Params: {} 2024-03-04T16:33:59.073 DEBUG:boto:Token: None 2024-03-04T16:33:59.073 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIUv15ztwXNHFf2vpWoP/brfvVo= 2024-03-04T16:33:59.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIUv15ztwXNHFf2vpWoP/brfvVo=', 'Content-Length': '0'} 2024-03-04T16:33:59.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004612015677e4f4f9-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.076 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.076 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.076 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.076 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.076 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.076 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.076 DEBUG:boto:Data: 2024-03-04T16:33:59.076 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.076 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.077 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.077 DEBUG:boto:Params: {} 2024-03-04T16:33:59.077 DEBUG:boto:Token: None 2024-03-04T16:33:59.077 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.077 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KIUv15ztwXNHFf2vpWoP/brfvVo= 2024-03-04T16:33:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KIUv15ztwXNHFf2vpWoP/brfvVo=', 'Content-Length': '0'} 2024-03-04T16:33:59.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3d4281c4bdd9782-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:59.079 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.079 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.079 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.079 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.080 DEBUG:boto:Method: GET 2024-03-04T16:33:59.080 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.080 DEBUG:boto:Data: 2024-03-04T16:33:59.080 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.080 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.080 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.080 DEBUG:boto:Params: {} 2024-03-04T16:33:59.080 DEBUG:boto:Token: None 2024-03-04T16:33:59.080 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.080 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4= 2024-03-04T16:33:59.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4=', 'Content-Length': '0'} 2024-03-04T16:33:59.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a81285198f793f75-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.124 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.124 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.124 DEBUG:boto:path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.124 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.125 DEBUG:boto:Method: GET 2024-03-04T16:33:59.125 DEBUG:boto:Path: /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.125 DEBUG:boto:Data: 2024-03-04T16:33:59.125 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.125 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.125 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.125 DEBUG:boto:Params: {} 2024-03-04T16:33:59.125 DEBUG:boto:Token: None 2024-03-04T16:33:59.125 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS 2024-03-04T16:33:59.126 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4= 2024-03-04T16:33:59.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6Ef3u1yPpcIW7IP6pO8LthD8Ks4=', 'Content-Length': '0'} 2024-03-04T16:33:59.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'Hclfl8Hokp6qeo82Mt5xCpcu3CPrjsS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016bb73647e3c2184-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.168 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.168 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.168 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.168 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.168 DEBUG:boto:Method: GET 2024-03-04T16:33:59.169 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.169 DEBUG:boto:Data: 2024-03-04T16:33:59.169 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.169 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.169 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.169 DEBUG:boto:Params: {} 2024-03-04T16:33:59.169 DEBUG:boto:Token: None 2024-03-04T16:33:59.169 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.169 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfc43c8e19c2591-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.175 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.176 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.176 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.176 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.176 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.176 DEBUG:boto:Method: GET 2024-03-04T16:33:59.176 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.176 DEBUG:boto:Data: 2024-03-04T16:33:59.176 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.176 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.176 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.177 DEBUG:boto:Params: {} 2024-03-04T16:33:59.177 DEBUG:boto:Token: None 2024-03-04T16:33:59.177 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.177 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e69b60c62b86c8d-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.180 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.180 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.180 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.181 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.181 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:59.181 DEBUG:boto:Data: 2024-03-04T16:33:59.181 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.181 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.181 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.181 DEBUG:boto:Params: {} 2024-03-04T16:33:59.181 DEBUG:boto:Token: None 2024-03-04T16:33:59.182 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj 2024-03-04T16:33:59.182 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU= 2024-03-04T16:33:59.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU=', 'Content-Length': '0'} 2024-03-04T16:33:59.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e917c00808255b8a-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.185 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.185 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.186 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.186 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:59.186 DEBUG:boto:Data: 2024-03-04T16:33:59.186 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.186 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.186 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.186 DEBUG:boto:Params: {} 2024-03-04T16:33:59.186 DEBUG:boto:Token: None 2024-03-04T16:33:59.186 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj 2024-03-04T16:33:59.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU= 2024-03-04T16:33:59.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU=', 'Content-Length': '0'} 2024-03-04T16:33:59.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e45e307e36315ca-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:59.190 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.190 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.190 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.190 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.190 DEBUG:boto:Method: GET 2024-03-04T16:33:59.190 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.191 DEBUG:boto:Data: 2024-03-04T16:33:59.191 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.191 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.191 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.191 DEBUG:boto:Params: {} 2024-03-04T16:33:59.191 DEBUG:boto:Token: None 2024-03-04T16:33:59.191 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.191 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y= 2024-03-04T16:33:59.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y=', 'Content-Length': '0'} 2024-03-04T16:33:59.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdfe4174a700ecf5-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.240 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.240 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.240 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.240 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.240 DEBUG:boto:Method: GET 2024-03-04T16:33:59.241 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.241 DEBUG:boto:Data: 2024-03-04T16:33:59.241 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.241 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.241 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.241 DEBUG:boto:Params: {} 2024-03-04T16:33:59.241 DEBUG:boto:Token: None 2024-03-04T16:33:59.241 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.241 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y= 2024-03-04T16:33:59.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y=', 'Content-Length': '0'} 2024-03-04T16:33:59.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6b67f6393f37574-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.244 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.244 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.244 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.244 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.244 DEBUG:boto:Method: GET 2024-03-04T16:33:59.244 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.244 DEBUG:boto:Data: 2024-03-04T16:33:59.244 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.244 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.245 DEBUG:boto:Params: {} 2024-03-04T16:33:59.245 DEBUG:boto:Token: None 2024-03-04T16:33:59.245 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f30ac3c0c35310b-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.296 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.297 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.297 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.297 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.297 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.297 DEBUG:boto:Method: GET 2024-03-04T16:33:59.297 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.297 DEBUG:boto:Data: 2024-03-04T16:33:59.297 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.297 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.298 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.298 DEBUG:boto:Params: {} 2024-03-04T16:33:59.298 DEBUG:boto:Token: None 2024-03-04T16:33:59.298 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80091695a968375-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.301 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:59.301 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.302 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.302 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.302 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.302 DEBUG:boto:Method: GET 2024-03-04T16:33:59.302 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.302 DEBUG:boto:Data: 2024-03-04T16:33:59.302 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.302 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.302 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.302 DEBUG:boto:Params: {} 2024-03-04T16:33:59.302 DEBUG:boto:Token: None 2024-03-04T16:33:59.303 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.303 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw= 2024-03-04T16:33:59.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw=', 'Content-Length': '0'} 2024-03-04T16:33:59.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046dd7a53d19f65eb-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.305 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.305 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.305 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.305 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.306 DEBUG:boto:Method: GET 2024-03-04T16:33:59.306 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.306 DEBUG:boto:Data: 2024-03-04T16:33:59.306 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.306 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.306 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.306 DEBUG:boto:Params: {} 2024-03-04T16:33:59.307 DEBUG:boto:Token: None 2024-03-04T16:33:59.307 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.307 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw= 2024-03-04T16:33:59.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw=', 'Content-Length': '0'} 2024-03-04T16:33:59.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045760c6c7c7c6ab5-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.352 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.352 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.352 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.352 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.352 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.353 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.353 DEBUG:boto:Data: 2024-03-04T16:33:59.353 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.353 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.353 DEBUG:boto:Params: {} 2024-03-04T16:33:59.353 DEBUG:boto:Token: None 2024-03-04T16:33:59.353 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:al33sLEIAUNMyxlEbzLOR3kuK/o= 2024-03-04T16:33:59.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:al33sLEIAUNMyxlEbzLOR3kuK/o=', 'Content-Length': '0'} 2024-03-04T16:33:59.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001898708f9daa9481-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.355 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.356 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.356 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.356 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.356 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.356 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.356 DEBUG:boto:Data: 2024-03-04T16:33:59.356 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.356 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.356 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.356 DEBUG:boto:Params: {} 2024-03-04T16:33:59.356 DEBUG:boto:Token: None 2024-03-04T16:33:59.357 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.357 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:al33sLEIAUNMyxlEbzLOR3kuK/o= 2024-03-04T16:33:59.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:al33sLEIAUNMyxlEbzLOR3kuK/o=', 'Content-Length': '0'} 2024-03-04T16:33:59.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0720662da3e4a63-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:59.359 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.359 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.359 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.359 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.360 DEBUG:boto:Method: GET 2024-03-04T16:33:59.360 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.360 DEBUG:boto:Data: 2024-03-04T16:33:59.360 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.360 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.360 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.360 DEBUG:boto:Params: {} 2024-03-04T16:33:59.360 DEBUG:boto:Token: None 2024-03-04T16:33:59.360 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.360 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw= 2024-03-04T16:33:59.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw=', 'Content-Length': '0'} 2024-03-04T16:33:59.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c83efb321ce3e6a-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.363 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.363 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.363 DEBUG:boto:path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.363 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.363 DEBUG:boto:Method: GET 2024-03-04T16:33:59.363 DEBUG:boto:Path: /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.363 DEBUG:boto:Data: 2024-03-04T16:33:59.363 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.363 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.363 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.363 DEBUG:boto:Params: {} 2024-03-04T16:33:59.364 DEBUG:boto:Token: None 2024-03-04T16:33:59.364 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH 2024-03-04T16:33:59.364 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw= 2024-03-04T16:33:59.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sj2D8BvTKNGhNZQeCkn90lNnjkw=', 'Content-Length': '0'} 2024-03-04T16:33:59.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:20 GMT'), ('x-amz-version-id', 'vsUdrKjgZ-TSKDZnFW0Jk3XJaxhhzFH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cdbfe1d6aa1f2df-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.408 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.408 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.408 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.409 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.409 DEBUG:boto:Method: GET 2024-03-04T16:33:59.409 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.409 DEBUG:boto:Data: 2024-03-04T16:33:59.409 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.409 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.409 DEBUG:boto:Params: {} 2024-03-04T16:33:59.409 DEBUG:boto:Token: None 2024-03-04T16:33:59.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d65d4897b33223b-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.412 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.413 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.413 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.413 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.413 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.413 DEBUG:boto:Method: GET 2024-03-04T16:33:59.413 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.413 DEBUG:boto:Data: 2024-03-04T16:33:59.413 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.413 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.413 DEBUG:boto:Params: {} 2024-03-04T16:33:59.414 DEBUG:boto:Token: None 2024-03-04T16:33:59.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.414 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd07411eda6f85a3-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.417 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.417 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.417 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.417 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.417 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:59.417 DEBUG:boto:Data: 2024-03-04T16:33:59.417 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.417 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.417 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.418 DEBUG:boto:Params: {} 2024-03-04T16:33:59.418 DEBUG:boto:Token: None 2024-03-04T16:33:59.418 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj 2024-03-04T16:33:59.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU= 2024-03-04T16:33:59.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU=', 'Content-Length': '0'} 2024-03-04T16:33:59.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008d740103400ff57-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.421 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.421 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.421 DEBUG:boto:Method: HEAD 2024-03-04T16:33:59.421 DEBUG:boto:Path: /sbnvju-22/obj 2024-03-04T16:33:59.421 DEBUG:boto:Data: 2024-03-04T16:33:59.421 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.421 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.421 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.422 DEBUG:boto:Params: {} 2024-03-04T16:33:59.422 DEBUG:boto:Token: None 2024-03-04T16:33:59.422 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj 2024-03-04T16:33:59.422 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU= 2024-03-04T16:33:59.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6xMBkfX8ad0P8ZJx4YF/XSk19MU=', 'Content-Length': '0'} 2024-03-04T16:33:59.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053843dd7cfe0c4b3-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:33:59.426 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.426 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.426 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.426 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.426 DEBUG:boto:Method: GET 2024-03-04T16:33:59.426 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.426 DEBUG:boto:Data: 2024-03-04T16:33:59.426 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.426 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.426 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.427 DEBUG:boto:Params: {} 2024-03-04T16:33:59.427 DEBUG:boto:Token: None 2024-03-04T16:33:59.427 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.427 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y= 2024-03-04T16:33:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y=', 'Content-Length': '0'} 2024-03-04T16:33:59.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-22:fc294220-5038-4f25-aeee-cac2c5838729.4341.31'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007af7710a2a60d2e9-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.472 DEBUG:boto:path=/sbnvju-22/obj 2024-03-04T16:33:59.472 DEBUG:boto:auth_path=/sbnvju-22/obj 2024-03-04T16:33:59.472 DEBUG:boto:path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.473 DEBUG:boto:auth_path=/sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.473 DEBUG:boto:Method: GET 2024-03-04T16:33:59.473 DEBUG:boto:Path: /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.473 DEBUG:boto:Data: 2024-03-04T16:33:59.473 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.473 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.473 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.473 DEBUG:boto:Params: {} 2024-03-04T16:33:59.473 DEBUG:boto:Token: None 2024-03-04T16:33:59.473 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/obj?versionId=7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi 2024-03-04T16:33:59.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y= 2024-03-04T16:33:59.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LbO9wrmBp6/YGQ995pXaCRZU49Y=', 'Content-Length': '0'} 2024-03-04T16:33:59.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:29:21 GMT'), ('x-amz-version-id', '7OSlhMe-S45YQ-C5o3I-Cprru4mMZYi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000728c4fe6e9740b83-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.476 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.476 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.476 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.476 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.477 DEBUG:boto:Method: GET 2024-03-04T16:33:59.477 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.477 DEBUG:boto:Data: 2024-03-04T16:33:59.477 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:33:59.477 DEBUG:boto:Port: 8001 2024-03-04T16:33:59.477 DEBUG:boto:Params: {} 2024-03-04T16:33:59.477 DEBUG:boto:Token: None 2024-03-04T16:33:59.477 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355c34b99ba2e426-0065e5f7f7-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.480 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.480 DEBUG:boto:path=/sbnvju-22/ 2024-03-04T16:33:59.481 DEBUG:boto:auth_path=/sbnvju-22/ 2024-03-04T16:33:59.481 DEBUG:boto:path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.481 DEBUG:boto:auth_path=/sbnvju-22/?prefix=obj 2024-03-04T16:33:59.481 DEBUG:boto:Method: GET 2024-03-04T16:33:59.481 DEBUG:boto:Path: /sbnvju-22/?prefix=obj 2024-03-04T16:33:59.481 DEBUG:boto:Data: 2024-03-04T16:33:59.481 DEBUG:boto:Headers: {} 2024-03-04T16:33:59.481 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:33:59.481 DEBUG:boto:Port: 8000 2024-03-04T16:33:59.481 DEBUG:boto:Params: {} 2024-03-04T16:33:59.481 DEBUG:boto:Token: None 2024-03-04T16:33:59.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:33:59 GMT /sbnvju-22/ 2024-03-04T16:33:59.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8= 2024-03-04T16:33:59.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:HfHVE6R0qFY1fWD0ypZ3jhcG/I8=', 'Content-Length': '0'} 2024-03-04T16:33:59.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20ad03e5fb29f52-0065e5f7f7-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:33:59.484 DEBUG:boto:b'sbnvju-22obj1000falseobj2024-03-04T16:29:21.440Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:33:59.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-22 zones={test-zone3, test-zone2} 2024-03-04T16:33:59.485 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-03-04T16:33:59.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.486 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:33:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:59.603 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"00000000000000000000:00000000000000009677","last_update":"2024-03-04T16:29:35.587546Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:41.563299Z"},{"marker":"00000000000000000000:00000000000000003755","last_update":"2024-03-04T16:25:55.578333Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T16:33:33.957864Z"}] 2024-03-04T16:33:59.604 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.604 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-03-04T16:33:59.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.604 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:33:59.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:59.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:59.730 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14474884151308337112},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:35.587546Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.563299Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:25:55.578333Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:33:33.957864Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:33:59Z"} 2024-03-04T16:33:59.730 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14474884151308337112},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:35.587546Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.563299Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:25:55.578333Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:33:33.957864Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:33:59Z"} 2024-03-04T16:33:59.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009677', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:35.587546Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:41.563299Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003755', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:25:55.578333Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:33:33.957864Z'}}] 2024-03-04T16:33:59.730 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.730 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.730 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-03-04T16:33:59.731 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-03-04T16:33:59.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.731 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:33:59.777 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:59.777 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:59.847 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"00000000000000000000:00000000000000009677","last_update":"2024-03-04T16:29:35.587546Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:41.563299Z"},{"marker":"00000000000000000000:00000000000000003755","last_update":"2024-03-04T16:25:55.578333Z"},{"marker":"00000000000000000000:00000000000000004319","last_update":"2024-03-04T16:33:33.957864Z"}] 2024-03-04T16:33:59.848 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.848 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-03-04T16:33:59.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.849 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:33:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:33:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:33:59.963 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9843214131471671687},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:35.587546Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.563299Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:25:55.578333Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:33:33.957864Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:33:59Z"} 2024-03-04T16:33:59.964 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9843214131471671687},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:35.587546Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.563299Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:25:55.578333Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004319","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:33:33.957864Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:33:59Z"} 2024-03-04T16:33:59.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009677', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:35.587546Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:41.563299Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003755', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:25:55.578333Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:33:33.957864Z'}}] 2024-03-04T16:33:59.964 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.964 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009677', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000003755', 3: '00000000000000000000:00000000000000004319'} 2024-03-04T16:33:59.964 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-03-04T16:33:59.964 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-03-04T16:33:59.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:33:59.965 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.080 INFO:teuthology.orchestra.run.smithi071.stdout:[{"marker":"00000000000000000000:00000000000000008549","last_update":"2024-03-04T16:28:51.615520Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:41.599468Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T16:27:42.523005Z"},{"marker":"00000000000000000000:00000000000000003755","last_update":"2024-03-04T16:29:57.619026Z"}] 2024-03-04T16:34:00.083 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.083 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-03-04T16:34:00.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.084 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.133 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.133 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.200 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2652177982598094693},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:28:51.615520Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.599468Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:27:42.523005Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:57.619026Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.201 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2652177982598094693},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:28:51.615520Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.599468Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:27:42.523005Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:57.619026Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008549', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:28:51.615520Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:41.599468Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:27:42.523005Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003755', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:57.619026Z'}}] 2024-03-04T16:34:00.201 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.201 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.201 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-03-04T16:34:00.201 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-03-04T16:34:00.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.202 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.247 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.312 INFO:teuthology.orchestra.run.smithi071.stdout:[{"marker":"00000000000000000000:00000000000000008549","last_update":"2024-03-04T16:28:51.615520Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:41.599468Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T16:27:42.523005Z"},{"marker":"00000000000000000000:00000000000000003755","last_update":"2024-03-04T16:29:57.619026Z"}] 2024-03-04T16:34:00.313 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.313 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-03-04T16:34:00.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.313 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.432 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9601393378351491655},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:28:51.615520Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.599468Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:27:42.523005Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:57.619026Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.432 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9601393378351491655},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:28:51.615520Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:41.599468Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:27:42.523005Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003755","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:57.619026Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008549', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:28:51.615520Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:41.599468Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:27:42.523005Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003755', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:57.619026Z'}}] 2024-03-04T16:34:00.432 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.433 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008549', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000003755'} 2024-03-04T16:34:00.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-03-04T16:34:00.433 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-03-04T16:34:00.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.433 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.549 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"00000000000000000000:00000000000000009395","last_update":"2024-03-04T16:29:19.693156Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:25.667511Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T16:26:01.687217Z"},{"marker":"00000000000000000000:00000000000000004037","last_update":"2024-03-04T16:31:00.629036Z"}] 2024-03-04T16:34:00.549 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.550 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-03-04T16:34:00.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.550 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.665 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6140546073098290770},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009395","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:19.693156Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:25.667511Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:26:01.687217Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:31:00.629036Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.665 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6140546073098290770},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009395","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:19.693156Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:25.667511Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:26:01.687217Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:31:00.629036Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009395', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:19.693156Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:25.667511Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:26:01.687217Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:31:00.629036Z'}}] 2024-03-04T16:34:00.666 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.666 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.666 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-03-04T16:34:00.666 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-03-04T16:34:00.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.666 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.782 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"00000000000000000000:00000000000000009395","last_update":"2024-03-04T16:29:19.693156Z"},{"marker":"00000000000000000000:00000000000000003760","last_update":"2024-03-04T16:19:25.667511Z"},{"marker":"00000000000000000000:00000000000000004178","last_update":"2024-03-04T16:26:01.687217Z"},{"marker":"00000000000000000000:00000000000000004037","last_update":"2024-03-04T16:31:00.629036Z"}] 2024-03-04T16:34:00.782 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.782 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-03-04T16:34:00.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.783 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.831 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.832 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:00.893 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6257827442330157336},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009395","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:19.693156Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:25.667511Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:26:01.687217Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:31:00.629036Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.894 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6257827442330157336},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009395","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:29:19.693156Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003760","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:19:25.667511Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004178","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:26:01.687217Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004037","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:31:00.629036Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-03-04T16:34:00Z"} 2024-03-04T16:34:00.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009395', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:29:19.693156Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003760', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:19:25.667511Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:26:01.687217Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:31:00.629036Z'}}] 2024-03-04T16:34:00.894 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.894 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009395', 1: '00000000000000000000:00000000000000003760', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000004037'} 2024-03-04T16:34:00.894 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-03-04T16:34:00.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-03-04T16:34:00.897 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:00.897 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:00.898 DEBUG:boto:Method: PUT 2024-03-04T16:34:00.898 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:00.898 DEBUG:boto:Data: 2024-03-04T16:34:00.898 DEBUG:boto:Headers: {} 2024-03-04T16:34:00.898 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:00.898 DEBUG:boto:Port: 8000 2024-03-04T16:34:00.898 DEBUG:boto:Params: {} 2024-03-04T16:34:00.898 DEBUG:boto:Token: None 2024-03-04T16:34:00.898 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:34:00 GMT /sbnvju-23/ 2024-03-04T16:34:00.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:INuX49psVsQK/bsnTpw1bJS8hM8= 2024-03-04T16:34:00.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:INuX49psVsQK/bsnTpw1bJS8hM8=', 'Content-Length': '0'} 2024-03-04T16:34:00.912 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000848340f1782eadc6-0065e5f7f8-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:00.912 DEBUG:rgw_multi.tests:created bucket=sbnvju-23 2024-03-04T16:34:00.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:00.913 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:34:01.024 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570040.919780_287.1","last_update":"2024-03-04T16:34:00.919780Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:34:01.025 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:01.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:34:01.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:01.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:01.025 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:01.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:01.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:01.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:01.123+0000 7fbc1529d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:01.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:01.124+0000 7fbc1529d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:01.133 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.913830_284.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.913830Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:01Z"} 2024-03-04T16:34:01.133 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.913830_284.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.913830Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:01Z"} 2024-03-04T16:34:01.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570040.913830_284.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:34:00.913830Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:34:01.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:01.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570040.913830_284.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:01.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570040.919780_287.1 target=1_1709570040.913830_284.1 2024-03-04T16:34:06.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:06.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:06.135 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:06.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:06.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:06.247+0000 7f757213b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:06.248+0000 7f757213b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:06.259 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.919780_287.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.919780Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:06Z"} 2024-03-04T16:34:06.259 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.919780_287.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.919780Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:06Z"} 2024-03-04T16:34:06.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570040.919780_287.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:34:00.919780Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:34:06.259 DEBUG:rgw_multi.tests:log_status={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:06.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:06.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:34:06.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:34:06.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:06.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:06.260 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:06.314 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:06.315 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:06.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:06.373+0000 7f7438b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:06.374+0000 7f7438b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:06.388 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.919780_287.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.919780Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:06Z"} 2024-03-04T16:34:06.389 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570040.919780_287.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:00.919780Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:06Z"} 2024-03-04T16:34:06.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570040.919780_287.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:34:00.919780Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:34:06.389 DEBUG:rgw_multi.tests:log_status={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:06.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570040.919780_287.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:06.389 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:34:06.389 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:06.389 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:06.390 DEBUG:boto:Method: HEAD 2024-03-04T16:34:06.390 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:06.390 DEBUG:boto:Data: 2024-03-04T16:34:06.390 DEBUG:boto:Headers: {} 2024-03-04T16:34:06.390 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:06.390 DEBUG:boto:Port: 8000 2024-03-04T16:34:06.390 DEBUG:boto:Params: {} 2024-03-04T16:34:06.390 DEBUG:boto:Token: None 2024-03-04T16:34:06.390 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:06 GMT /sbnvju-23/ 2024-03-04T16:34:06.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OMq/SbWCWMSoYv+GfSdGb+7TVlI= 2024-03-04T16:34:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OMq/SbWCWMSoYv+GfSdGb+7TVlI=', 'Content-Length': '0'} 2024-03-04T16:34:06.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f1b64288251c11-0065e5f7fe-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:06.394 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:06.394 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:06.394 DEBUG:boto:Method: PUT 2024-03-04T16:34:06.394 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:06.394 DEBUG:boto:Data: 2024-03-04T16:34:06.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:34:06.394 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:06.394 DEBUG:boto:Port: 8000 2024-03-04T16:34:06.394 DEBUG:boto:Params: {} 2024-03-04T16:34:06.394 DEBUG:boto:Token: None 2024-03-04T16:34:06.395 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:34:06 GMT /sbnvju-23/obj 2024-03-04T16:34:06.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6FvKzhvHENg0p3/JaCMXK5g60wI= 2024-03-04T16:34:06.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:34:06 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6FvKzhvHENg0p3/JaCMXK5g60wI='} 2024-03-04T16:34:06.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077d363154ed067de-0065e5f7fe-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:06.401 DEBUG:rgw_multi.tests:created initial version id=None 2024-03-04T16:34:06.401 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:34:06.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:06.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:06.402 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:06.456 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:06.456 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:06.503 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:06.513+0000 7f0874224800 1 waiting to reach incremental sync.. 2024-03-04T16:34:11.504 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:11.512+0000 7f0874224800 1 waiting to reach incremental sync.. 2024-03-04T16:34:16.504 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:16.512+0000 7f0874224800 1 waiting to reach incremental sync.. 2024-03-04T16:34:21.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:21.514+0000 7f0874224800 1 bucket sync caught up with source: 2024-03-04T16:34:21.506 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:21.506 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:21.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:21.514+0000 7f0874224800 0 bucket checkpoint complete 2024-03-04T16:34:21.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:34:21.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:21.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:21.530 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:21.595 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:21.595 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:21.658+0000 7fe27de0f800 1 waiting to reach incremental sync.. 2024-03-04T16:34:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:26.660+0000 7fe27de0f800 1 bucket sync caught up with source: 2024-03-04T16:34:26.652 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:26.652 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:26.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:26.660+0000 7fe27de0f800 0 bucket checkpoint complete 2024-03-04T16:34:26.675 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:34:26.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:26.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:26.675 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:26.733 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:26.796+0000 7f3583239800 1 waiting to reach incremental sync.. 2024-03-04T16:34:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:31.797+0000 7f3583239800 1 waiting to reach incremental sync.. 2024-03-04T16:34:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:36.799+0000 7f3583239800 1 bucket sync caught up with source: 2024-03-04T16:34:36.790 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:36.790 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:36.799+0000 7f3583239800 0 bucket checkpoint complete 2024-03-04T16:34:36.814 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:34:36.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:36.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:36.815 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:36.874 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:36.937+0000 7f133e91f800 1 bucket sync caught up with source: 2024-03-04T16:34:36.929 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:36.929 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:36.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:36.937+0000 7f133e91f800 0 bucket checkpoint complete 2024-03-04T16:34:36.947 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:34:36.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:36.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:36.948 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:37.002 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:37.002 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:37.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:37.061+0000 7fb9ccc0f800 1 bucket sync caught up with source: 2024-03-04T16:34:37.053 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:37.053 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:37.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:37.061+0000 7fb9ccc0f800 0 bucket checkpoint complete 2024-03-04T16:34:37.071 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:34:37.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:37.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:37.072 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:37.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:37.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:37.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:37.197+0000 7f7c4d417800 1 bucket sync caught up with source: 2024-03-04T16:34:37.188 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:37.188 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:37.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:37.197+0000 7f7c4d417800 0 bucket checkpoint complete 2024-03-04T16:34:37.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:34:37.207 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.207 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.207 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.207 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.207 DEBUG:boto:Data: 2024-03-04T16:34:37.207 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.208 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.208 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.208 DEBUG:boto:Params: {} 2024-03-04T16:34:37.208 DEBUG:boto:Token: None 2024-03-04T16:34:37.208 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.208 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef23cd5af272b4bc-0065e5f81d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.211 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.211 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.211 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.212 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.212 DEBUG:boto:Data: 2024-03-04T16:34:37.212 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.212 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.212 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.212 DEBUG:boto:Params: {} 2024-03-04T16:34:37.212 DEBUG:boto:Token: None 2024-03-04T16:34:37.212 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.212 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce185f4e8bd41c14-0065e5f81d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:34:37.215 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.215 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.215 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.215 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.215 DEBUG:boto:Method: GET 2024-03-04T16:34:37.215 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.216 DEBUG:boto:Data: 2024-03-04T16:34:37.216 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.216 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.216 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.216 DEBUG:boto:Params: {} 2024-03-04T16:34:37.216 DEBUG:boto:Token: None 2024-03-04T16:34:37.216 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.216 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000830603c673adbb2b-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.219 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.220 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:34:37.220 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.220 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.220 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.220 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.220 DEBUG:boto:Method: GET 2024-03-04T16:34:37.220 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.220 DEBUG:boto:Data: 2024-03-04T16:34:37.220 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.221 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.221 DEBUG:boto:Params: {} 2024-03-04T16:34:37.221 DEBUG:boto:Token: None 2024-03-04T16:34:37.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.221 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b196fb63184b16f7-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.224 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.224 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.225 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.225 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.225 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.225 DEBUG:boto:Method: GET 2024-03-04T16:34:37.225 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.225 DEBUG:boto:Data: 2024-03-04T16:34:37.225 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.225 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.225 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.225 DEBUG:boto:Params: {} 2024-03-04T16:34:37.226 DEBUG:boto:Token: None 2024-03-04T16:34:37.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360d5f52f7a5c1bb-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.229 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.229 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.229 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.229 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.230 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.230 DEBUG:boto:Method: GET 2024-03-04T16:34:37.230 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.230 DEBUG:boto:Data: 2024-03-04T16:34:37.230 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.230 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.230 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.230 DEBUG:boto:Params: {} 2024-03-04T16:34:37.230 DEBUG:boto:Token: None 2024-03-04T16:34:37.230 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.230 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5cd6e4f4d12230-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.233 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.234 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.234 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.234 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.234 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.234 DEBUG:boto:Method: GET 2024-03-04T16:34:37.234 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.234 DEBUG:boto:Data: 2024-03-04T16:34:37.234 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.234 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.235 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.235 DEBUG:boto:Params: {} 2024-03-04T16:34:37.235 DEBUG:boto:Token: None 2024-03-04T16:34:37.235 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.235 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ad0418fb1c0b550-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.237 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.237 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.238 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.238 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.238 DEBUG:boto:Method: GET 2024-03-04T16:34:37.238 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.238 DEBUG:boto:Data: 2024-03-04T16:34:37.238 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.238 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.238 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.238 DEBUG:boto:Params: {} 2024-03-04T16:34:37.238 DEBUG:boto:Token: None 2024-03-04T16:34:37.238 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.239 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0b66e0fcf11779c-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.241 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.241 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.241 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.241 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.241 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.241 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.241 DEBUG:boto:Data: 2024-03-04T16:34:37.241 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.242 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.242 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.242 DEBUG:boto:Params: {} 2024-03-04T16:34:37.242 DEBUG:boto:Token: None 2024-03-04T16:34:37.242 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.242 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bf6a7042cc4a7ab-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.244 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.244 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.244 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.244 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.245 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.245 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.245 DEBUG:boto:Data: 2024-03-04T16:34:37.245 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.245 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.245 DEBUG:boto:Params: {} 2024-03-04T16:34:37.245 DEBUG:boto:Token: None 2024-03-04T16:34:37.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1f64d735ad6ef8f-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.247 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.248 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.248 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.248 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.248 DEBUG:boto:Method: GET 2024-03-04T16:34:37.248 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.248 DEBUG:boto:Data: 2024-03-04T16:34:37.248 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.248 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.249 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.249 DEBUG:boto:Params: {} 2024-03-04T16:34:37.249 DEBUG:boto:Token: None 2024-03-04T16:34:37.249 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c092bea241fb56d8-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.251 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.251 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.251 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.251 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.251 DEBUG:boto:Method: GET 2024-03-04T16:34:37.252 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.252 DEBUG:boto:Data: 2024-03-04T16:34:37.252 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.252 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.252 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.252 DEBUG:boto:Params: {} 2024-03-04T16:34:37.252 DEBUG:boto:Token: None 2024-03-04T16:34:37.252 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.252 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c1a0e3b54d2eaf3-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.254 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.254 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.255 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.255 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.255 DEBUG:boto:Method: GET 2024-03-04T16:34:37.255 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.255 DEBUG:boto:Data: 2024-03-04T16:34:37.255 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.255 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.255 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.255 DEBUG:boto:Params: {} 2024-03-04T16:34:37.255 DEBUG:boto:Token: None 2024-03-04T16:34:37.255 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.256 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2c957e7f66c7e4-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.304 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.305 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.305 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.305 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.305 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.305 DEBUG:boto:Method: GET 2024-03-04T16:34:37.305 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.305 DEBUG:boto:Data: 2024-03-04T16:34:37.305 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.305 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.305 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.305 DEBUG:boto:Params: {} 2024-03-04T16:34:37.306 DEBUG:boto:Token: None 2024-03-04T16:34:37.306 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.306 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e1934c0f98412f-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.308 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.309 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.309 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.309 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.309 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.309 DEBUG:boto:Data: 2024-03-04T16:34:37.309 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.309 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.309 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.309 DEBUG:boto:Params: {} 2024-03-04T16:34:37.310 DEBUG:boto:Token: None 2024-03-04T16:34:37.310 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.310 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051596a41c539d71e-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.312 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.313 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.313 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.313 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.313 DEBUG:boto:Data: 2024-03-04T16:34:37.313 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.313 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.313 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.313 DEBUG:boto:Params: {} 2024-03-04T16:34:37.313 DEBUG:boto:Token: None 2024-03-04T16:34:37.313 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e49466aa038312e-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.316 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.316 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.316 DEBUG:boto:Method: GET 2024-03-04T16:34:37.316 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.316 DEBUG:boto:Data: 2024-03-04T16:34:37.316 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.316 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.316 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.316 DEBUG:boto:Params: {} 2024-03-04T16:34:37.317 DEBUG:boto:Token: None 2024-03-04T16:34:37.317 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.317 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b972b62f4e6e3a0-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.319 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.319 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.319 DEBUG:boto:Method: GET 2024-03-04T16:34:37.319 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.319 DEBUG:boto:Data: 2024-03-04T16:34:37.319 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.319 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.319 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.320 DEBUG:boto:Params: {} 2024-03-04T16:34:37.320 DEBUG:boto:Token: None 2024-03-04T16:34:37.320 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.320 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4a776c9ca3428d3-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.322 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.322 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.322 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.322 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.322 DEBUG:boto:Method: GET 2024-03-04T16:34:37.322 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.322 DEBUG:boto:Data: 2024-03-04T16:34:37.322 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.323 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.323 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.323 DEBUG:boto:Params: {} 2024-03-04T16:34:37.323 DEBUG:boto:Token: None 2024-03-04T16:34:37.323 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.323 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb299a3c43f70cb1-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.325 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.325 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.326 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.326 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.326 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.326 DEBUG:boto:Method: GET 2024-03-04T16:34:37.326 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.326 DEBUG:boto:Data: 2024-03-04T16:34:37.326 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.326 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.327 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.327 DEBUG:boto:Params: {} 2024-03-04T16:34:37.327 DEBUG:boto:Token: None 2024-03-04T16:34:37.327 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.327 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000053735204225023f-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.330 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:34:37.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:34:37.330 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.330 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.330 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.330 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.330 DEBUG:boto:Data: 2024-03-04T16:34:37.331 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.331 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.331 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.331 DEBUG:boto:Params: {} 2024-03-04T16:34:37.331 DEBUG:boto:Token: None 2024-03-04T16:34:37.331 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.331 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000914da5e7bc5a0da6-0065e5f81d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.334 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.334 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.334 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.335 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.335 DEBUG:boto:Data: 2024-03-04T16:34:37.335 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.335 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.335 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.335 DEBUG:boto:Params: {} 2024-03-04T16:34:37.335 DEBUG:boto:Token: None 2024-03-04T16:34:37.335 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.335 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081f6ea95eb6ca090-0065e5f81d-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:34:37.338 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.338 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.338 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.338 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.338 DEBUG:boto:Method: GET 2024-03-04T16:34:37.338 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.338 DEBUG:boto:Data: 2024-03-04T16:34:37.338 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.338 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.339 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.339 DEBUG:boto:Params: {} 2024-03-04T16:34:37.339 DEBUG:boto:Token: None 2024-03-04T16:34:37.339 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.339 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001151461e185ff215-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.341 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:34:37.342 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.342 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.342 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.342 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.342 DEBUG:boto:Method: GET 2024-03-04T16:34:37.342 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.343 DEBUG:boto:Data: 2024-03-04T16:34:37.343 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.343 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.343 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.343 DEBUG:boto:Params: {} 2024-03-04T16:34:37.343 DEBUG:boto:Token: None 2024-03-04T16:34:37.343 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.343 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506e7f7acb2e23a7-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.345 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.346 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.346 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.346 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.346 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.346 DEBUG:boto:Method: GET 2024-03-04T16:34:37.346 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.346 DEBUG:boto:Data: 2024-03-04T16:34:37.347 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.347 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.347 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.347 DEBUG:boto:Params: {} 2024-03-04T16:34:37.347 DEBUG:boto:Token: None 2024-03-04T16:34:37.347 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.347 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff02ffc63fc45a97-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.349 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.350 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.350 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.350 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.350 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.350 DEBUG:boto:Method: GET 2024-03-04T16:34:37.350 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.350 DEBUG:boto:Data: 2024-03-04T16:34:37.350 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.350 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.350 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.351 DEBUG:boto:Params: {} 2024-03-04T16:34:37.351 DEBUG:boto:Token: None 2024-03-04T16:34:37.351 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.351 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c1765dd38119d2-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.400 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.401 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.401 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.401 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.401 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.401 DEBUG:boto:Method: GET 2024-03-04T16:34:37.402 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.402 DEBUG:boto:Data: 2024-03-04T16:34:37.402 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.402 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.402 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.402 DEBUG:boto:Params: {} 2024-03-04T16:34:37.402 DEBUG:boto:Token: None 2024-03-04T16:34:37.402 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ec2cffff4c070ca-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.405 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.405 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.405 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.405 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.405 DEBUG:boto:Method: GET 2024-03-04T16:34:37.405 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.405 DEBUG:boto:Data: 2024-03-04T16:34:37.405 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.405 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.405 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.405 DEBUG:boto:Params: {} 2024-03-04T16:34:37.406 DEBUG:boto:Token: None 2024-03-04T16:34:37.406 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c9b2e430d8a0422-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.408 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.408 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.408 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.408 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.408 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.409 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.409 DEBUG:boto:Data: 2024-03-04T16:34:37.409 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.409 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.409 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.409 DEBUG:boto:Params: {} 2024-03-04T16:34:37.409 DEBUG:boto:Token: None 2024-03-04T16:34:37.409 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a606fe7a9240f666-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.411 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.411 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.411 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.411 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.411 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.412 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.412 DEBUG:boto:Data: 2024-03-04T16:34:37.412 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.412 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.412 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.412 DEBUG:boto:Params: {} 2024-03-04T16:34:37.412 DEBUG:boto:Token: None 2024-03-04T16:34:37.412 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.412 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007faf06e2f91ff1d4-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.414 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.415 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.415 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.415 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.415 DEBUG:boto:Method: GET 2024-03-04T16:34:37.415 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.415 DEBUG:boto:Data: 2024-03-04T16:34:37.415 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.415 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.415 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.415 DEBUG:boto:Params: {} 2024-03-04T16:34:37.415 DEBUG:boto:Token: None 2024-03-04T16:34:37.416 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.416 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af01ac6471b60340-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.417 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.418 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.418 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.418 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.418 DEBUG:boto:Method: GET 2024-03-04T16:34:37.418 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.418 DEBUG:boto:Data: 2024-03-04T16:34:37.418 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.418 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.418 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.418 DEBUG:boto:Params: {} 2024-03-04T16:34:37.418 DEBUG:boto:Token: None 2024-03-04T16:34:37.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2e2cbcac3db94f4-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.420 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.420 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.420 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.420 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.421 DEBUG:boto:Method: GET 2024-03-04T16:34:37.421 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.421 DEBUG:boto:Data: 2024-03-04T16:34:37.421 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.421 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.421 DEBUG:boto:Params: {} 2024-03-04T16:34:37.421 DEBUG:boto:Token: None 2024-03-04T16:34:37.421 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f277bdb15cb274a7-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.468 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.469 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.469 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.469 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.469 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.469 DEBUG:boto:Method: GET 2024-03-04T16:34:37.469 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.469 DEBUG:boto:Data: 2024-03-04T16:34:37.469 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.469 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.469 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.470 DEBUG:boto:Params: {} 2024-03-04T16:34:37.470 DEBUG:boto:Token: None 2024-03-04T16:34:37.470 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.470 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af8d460c09b124ba-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.473 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.473 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.473 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.473 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.473 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.473 DEBUG:boto:Data: 2024-03-04T16:34:37.473 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.474 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.474 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.474 DEBUG:boto:Params: {} 2024-03-04T16:34:37.474 DEBUG:boto:Token: None 2024-03-04T16:34:37.474 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec75d4032590968c-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.476 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.477 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.477 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.477 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.477 DEBUG:boto:Data: 2024-03-04T16:34:37.477 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.477 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.477 DEBUG:boto:Params: {} 2024-03-04T16:34:37.477 DEBUG:boto:Token: None 2024-03-04T16:34:37.478 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af8d1abb737a6df5-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.479 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.480 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.480 DEBUG:boto:Method: GET 2024-03-04T16:34:37.480 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.480 DEBUG:boto:Data: 2024-03-04T16:34:37.480 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.480 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.480 DEBUG:boto:Params: {} 2024-03-04T16:34:37.480 DEBUG:boto:Token: None 2024-03-04T16:34:37.481 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.481 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033d63c331cbc3835-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.483 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.483 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.483 DEBUG:boto:Method: GET 2024-03-04T16:34:37.483 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.483 DEBUG:boto:Data: 2024-03-04T16:34:37.483 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.483 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.483 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.483 DEBUG:boto:Params: {} 2024-03-04T16:34:37.483 DEBUG:boto:Token: None 2024-03-04T16:34:37.484 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.484 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8eac672976dc60e-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.486 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.486 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.486 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.486 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.486 DEBUG:boto:Method: GET 2024-03-04T16:34:37.486 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.486 DEBUG:boto:Data: 2024-03-04T16:34:37.486 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.486 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.487 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.487 DEBUG:boto:Params: {} 2024-03-04T16:34:37.487 DEBUG:boto:Token: None 2024-03-04T16:34:37.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.487 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254d12ab38d58ffc-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.490 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.490 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.490 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.490 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.490 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.491 DEBUG:boto:Method: GET 2024-03-04T16:34:37.491 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.491 DEBUG:boto:Data: 2024-03-04T16:34:37.491 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.491 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.491 DEBUG:boto:Params: {} 2024-03-04T16:34:37.491 DEBUG:boto:Token: None 2024-03-04T16:34:37.491 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a952916c3266ba1-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.494 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.494 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:34:37.494 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:34:37.494 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.494 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.495 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.495 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.495 DEBUG:boto:Data: 2024-03-04T16:34:37.495 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.495 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.495 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.495 DEBUG:boto:Params: {} 2024-03-04T16:34:37.495 DEBUG:boto:Token: None 2024-03-04T16:34:37.495 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.496 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091e1f7decd2b24a4-0065e5f81d-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.498 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.498 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.498 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.498 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:37.498 DEBUG:boto:Data: 2024-03-04T16:34:37.498 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.499 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.499 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.499 DEBUG:boto:Params: {} 2024-03-04T16:34:37.499 DEBUG:boto:Token: None 2024-03-04T16:34:37.499 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.499 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM= 2024-03-04T16:34:37.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7ZvxaQzC53ynrGRUotZ9I29BFgM=', 'Content-Length': '0'} 2024-03-04T16:34:37.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b20f09aa2a4d8cf8-0065e5f81d-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:34:37.505 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.505 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.505 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.505 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.505 DEBUG:boto:Method: GET 2024-03-04T16:34:37.505 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.505 DEBUG:boto:Data: 2024-03-04T16:34:37.505 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.505 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.505 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.506 DEBUG:boto:Params: {} 2024-03-04T16:34:37.506 DEBUG:boto:Token: None 2024-03-04T16:34:37.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d6a842ea2a6588-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.508 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.509 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:34:37.509 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.509 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.509 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.509 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.509 DEBUG:boto:Method: GET 2024-03-04T16:34:37.509 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.509 DEBUG:boto:Data: 2024-03-04T16:34:37.509 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.510 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.510 DEBUG:boto:Params: {} 2024-03-04T16:34:37.510 DEBUG:boto:Token: None 2024-03-04T16:34:37.510 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac58173ae56d92d-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.556 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:34:37.557 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.557 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.557 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.557 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.557 DEBUG:boto:Method: GET 2024-03-04T16:34:37.557 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.557 DEBUG:boto:Data: 2024-03-04T16:34:37.557 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.557 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.557 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.558 DEBUG:boto:Params: {} 2024-03-04T16:34:37.558 DEBUG:boto:Token: None 2024-03-04T16:34:37.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000445ed414c9f11826-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.561 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.561 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.561 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.561 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.561 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.561 DEBUG:boto:Method: GET 2024-03-04T16:34:37.561 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:34:37.562 DEBUG:boto:Data: 2024-03-04T16:34:37.562 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.562 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.562 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.562 DEBUG:boto:Params: {} 2024-03-04T16:34:37.562 DEBUG:boto:Token: None 2024-03-04T16:34:37.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versions 2024-03-04T16:34:37.563 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I= 2024-03-04T16:34:37.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lmQTBjSslJ2lCFFfUYoExVgJQ5I=', 'Content-Length': '0'} 2024-03-04T16:34:37.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a1d69dc79adc1db-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.565 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.566 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.566 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.566 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.566 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.566 DEBUG:boto:Method: GET 2024-03-04T16:34:37.566 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.566 DEBUG:boto:Data: 2024-03-04T16:34:37.566 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.566 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.566 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.566 DEBUG:boto:Params: {} 2024-03-04T16:34:37.567 DEBUG:boto:Token: None 2024-03-04T16:34:37.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c4273328c2a7f82-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.569 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.569 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.569 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.570 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.570 DEBUG:boto:Method: GET 2024-03-04T16:34:37.570 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.570 DEBUG:boto:Data: 2024-03-04T16:34:37.570 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.570 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.570 DEBUG:boto:Params: {} 2024-03-04T16:34:37.570 DEBUG:boto:Token: None 2024-03-04T16:34:37.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cf35215e158624c-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.573 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.573 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.573 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.573 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.573 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.573 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.573 DEBUG:boto:Data: 2024-03-04T16:34:37.573 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.574 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.574 DEBUG:boto:Params: {} 2024-03-04T16:34:37.574 DEBUG:boto:Token: None 2024-03-04T16:34:37.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000441cbf8ee6569726-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.576 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.576 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.576 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.576 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.576 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.577 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.577 DEBUG:boto:Data: 2024-03-04T16:34:37.577 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.577 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.577 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.577 DEBUG:boto:Params: {} 2024-03-04T16:34:37.577 DEBUG:boto:Token: None 2024-03-04T16:34:37.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI= 2024-03-04T16:34:37.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EQJ9gSUAbO2wzeeRnlKdcoPqGpI=', 'Content-Length': '0'} 2024-03-04T16:34:37.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000955a00cf3a70cf38-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.579 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.580 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.580 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.580 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.580 DEBUG:boto:Method: GET 2024-03-04T16:34:37.580 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.580 DEBUG:boto:Data: 2024-03-04T16:34:37.580 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.580 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.581 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.581 DEBUG:boto:Params: {} 2024-03-04T16:34:37.581 DEBUG:boto:Token: None 2024-03-04T16:34:37.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db0b9f1443c6ea99-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.584 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.584 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.584 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.584 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.584 DEBUG:boto:Method: GET 2024-03-04T16:34:37.585 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.585 DEBUG:boto:Data: 2024-03-04T16:34:37.585 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.585 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.585 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.585 DEBUG:boto:Params: {} 2024-03-04T16:34:37.586 DEBUG:boto:Token: None 2024-03-04T16:34:37.586 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:34:37.586 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0= 2024-03-04T16:34:37.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P9kzPR9iqjK6fd+eIJXZZtADJC0=', 'Content-Length': '0'} 2024-03-04T16:34:37.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c7b1e083bf66d6a-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.588 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.589 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.589 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.589 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.589 DEBUG:boto:Method: GET 2024-03-04T16:34:37.589 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.590 DEBUG:boto:Data: 2024-03-04T16:34:37.590 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.590 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.590 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.590 DEBUG:boto:Params: {} 2024-03-04T16:34:37.590 DEBUG:boto:Token: None 2024-03-04T16:34:37.590 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.590 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b36c56b44dd2c44-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.640 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.640 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.641 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.641 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.641 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.641 DEBUG:boto:Method: GET 2024-03-04T16:34:37.641 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.641 DEBUG:boto:Data: 2024-03-04T16:34:37.641 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.641 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.641 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.642 DEBUG:boto:Params: {} 2024-03-04T16:34:37.642 DEBUG:boto:Token: None 2024-03-04T16:34:37.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6b8aedd331676e-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.645 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.645 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.646 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.646 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.646 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.646 DEBUG:boto:Data: 2024-03-04T16:34:37.646 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.646 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.646 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.646 DEBUG:boto:Params: {} 2024-03-04T16:34:37.646 DEBUG:boto:Token: None 2024-03-04T16:34:37.646 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000476ec9a05734781d-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.648 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.648 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.649 DEBUG:boto:Method: HEAD 2024-03-04T16:34:37.649 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.649 DEBUG:boto:Data: 2024-03-04T16:34:37.649 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.649 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.649 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.649 DEBUG:boto:Params: {} 2024-03-04T16:34:37.649 DEBUG:boto:Token: None 2024-03-04T16:34:37.649 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ= 2024-03-04T16:34:37.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1TLWdm6ewWhjDuc2x4qNTHbQ3AQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e4cce1d2facbfcd1-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:34:37.651 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.651 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.652 DEBUG:boto:Method: GET 2024-03-04T16:34:37.652 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.652 DEBUG:boto:Data: 2024-03-04T16:34:37.652 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.652 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.652 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.652 DEBUG:boto:Params: {} 2024-03-04T16:34:37.652 DEBUG:boto:Token: None 2024-03-04T16:34:37.652 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.652 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005bae0a468f927083-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.657 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:37.657 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:37.657 DEBUG:boto:Method: GET 2024-03-04T16:34:37.657 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:37.657 DEBUG:boto:Data: 2024-03-04T16:34:37.657 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.657 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.657 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.657 DEBUG:boto:Params: {} 2024-03-04T16:34:37.657 DEBUG:boto:Token: None 2024-03-04T16:34:37.658 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/obj 2024-03-04T16:34:37.658 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ= 2024-03-04T16:34:37.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3vzfRd6l23rzYWDuguS/mRo3jEQ=', 'Content-Length': '0'} 2024-03-04T16:34:37.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054ec3d1def916260-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.659 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.660 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.660 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.660 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.660 DEBUG:boto:Method: GET 2024-03-04T16:34:37.660 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.660 DEBUG:boto:Data: 2024-03-04T16:34:37.660 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:34:37.660 DEBUG:boto:Port: 8001 2024-03-04T16:34:37.660 DEBUG:boto:Params: {} 2024-03-04T16:34:37.660 DEBUG:boto:Token: None 2024-03-04T16:34:37.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.661 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d457305bfef678e3-0065e5f81d-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.704 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.705 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.705 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.705 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.705 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:34:37.705 DEBUG:boto:Method: GET 2024-03-04T16:34:37.705 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:34:37.705 DEBUG:boto:Data: 2024-03-04T16:34:37.705 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.705 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:34:37.705 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.706 DEBUG:boto:Params: {} 2024-03-04T16:34:37.706 DEBUG:boto:Token: None 2024-03-04T16:34:37.706 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/ 2024-03-04T16:34:37.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo= 2024-03-04T16:34:37.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cx7Nz0O4uQGIgSe4LEe5rvAEWxo=', 'Content-Length': '0'} 2024-03-04T16:34:37.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9700e029192b355-0065e5f81d-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.709 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:34:37.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:34:37.709 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:37.709 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:37.709 DEBUG:boto:path=/sbnvju-23/?versioning 2024-03-04T16:34:37.709 DEBUG:boto:auth_path=/sbnvju-23/?versioning 2024-03-04T16:34:37.709 DEBUG:boto:Method: PUT 2024-03-04T16:34:37.709 DEBUG:boto:Path: /sbnvju-23/?versioning 2024-03-04T16:34:37.710 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:34:37.710 DEBUG:boto:Headers: {} 2024-03-04T16:34:37.710 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:37.710 DEBUG:boto:Port: 8000 2024-03-04T16:34:37.710 DEBUG:boto:Params: {} 2024-03-04T16:34:37.710 DEBUG:boto:Token: None 2024-03-04T16:34:37.710 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:34:37 GMT /sbnvju-23/?versioning 2024-03-04T16:34:37.710 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UCFUsq6orgtvPnRsloBb6I0owr0= 2024-03-04T16:34:37.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UCFUsq6orgtvPnRsloBb6I0owr0=', 'Content-Length': '228'} 2024-03-04T16:34:37.716 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072dea609235457ab-0065e5f81d-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:37.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.716 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:34:37.834 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570077.725329_305.1","last_update":"2024-03-04T16:34:37.725329Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:34:37.834 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570077.725329_305.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:37.834 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:34:37.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.835 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:37.923 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:37.932+0000 7ff13f133800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:37.924 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:37.933+0000 7ff13f133800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:37.942 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570077.725329_305.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:37.725329Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:37Z"} 2024-03-04T16:34:37.942 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570077.725329_305.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:37.725329Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:37Z"} 2024-03-04T16:34:37.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570077.725329_305.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:34:37.725329Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:34:37.943 DEBUG:rgw_multi.tests:log_status={0: '1_1709570077.725329_305.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:37.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570077.725329_305.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:37.944 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:34:37.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:34:37.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:34:37.945 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:34:37.993 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:38.047+0000 7f767eb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:38.048+0000 7f767eb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:34:38.058 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570077.725329_305.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:37.725329Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:38Z"} 2024-03-04T16:34:38.059 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570077.725329_305.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:34:37.725329Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:34:38Z"} 2024-03-04T16:34:38.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570077.725329_305.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:34:37.725329Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:34:38.059 DEBUG:rgw_multi.tests:log_status={0: '1_1709570077.725329_305.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:38.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570077.725329_305.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:34:38.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:34:38.060 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:34:38.060 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:34:38.060 DEBUG:boto:Method: HEAD 2024-03-04T16:34:38.060 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:34:38.060 DEBUG:boto:Data: 2024-03-04T16:34:38.060 DEBUG:boto:Headers: {} 2024-03-04T16:34:38.060 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:38.060 DEBUG:boto:Port: 8000 2024-03-04T16:34:38.060 DEBUG:boto:Params: {} 2024-03-04T16:34:38.061 DEBUG:boto:Token: None 2024-03-04T16:34:38.061 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:34:38 GMT /sbnvju-23/ 2024-03-04T16:34:38.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EJuJgm6xPf6HXyFZHckBjS8soMc= 2024-03-04T16:34:38.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:34:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EJuJgm6xPf6HXyFZHckBjS8soMc=', 'Content-Length': '0'} 2024-03-04T16:34:38.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a454830e3f80c1cb-0065e5f81e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:38.064 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:34:38.064 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:34:38.065 DEBUG:boto:Method: PUT 2024-03-04T16:34:38.065 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:34:38.065 DEBUG:boto:Data: 2024-03-04T16:34:38.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:34:38.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:34:38.065 DEBUG:boto:Port: 8000 2024-03-04T16:34:38.065 DEBUG:boto:Params: {} 2024-03-04T16:34:38.065 DEBUG:boto:Token: None 2024-03-04T16:34:38.065 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:34:38 GMT /sbnvju-23/obj 2024-03-04T16:34:38.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Kvlfgct/Xeqx95R/FrEkUweYoCo= 2024-03-04T16:34:38.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:34:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Kvlfgct/Xeqx95R/FrEkUweYoCo='} 2024-03-04T16:34:38.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-amz-request-id', 'tx000002fb580ef2b724651-0065e5f81e-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:34:38.075 DEBUG:rgw_multi.tests:created new version id=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:34:38.075 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:34:38.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:38.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:38.076 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:38.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:38.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:38.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:38.185+0000 7f967aa2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:34:38.176 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:38.177 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:43.187+0000 7f967aa2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:34:43.178 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:43.178 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:48.179 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:48.188+0000 7f967aa2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:34:48.179 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:48.179 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:53.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:53.189+0000 7f967aa2e800 1 waiting for incremental sync to catch up: 2024-03-04T16:34:53.180 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2024-03-04T16:34:53.181 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:58.181 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:58.191+0000 7f967aa2e800 1 bucket sync caught up with source: 2024-03-04T16:34:58.182 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:58.182 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:58.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:34:58.191+0000 7f967aa2e800 0 bucket checkpoint complete 2024-03-04T16:34:58.205 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:34:58.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:58.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:58.205 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:58.267 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:58.268 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:58.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:58.332+0000 7fbbae00f800 1 bucket sync caught up with source: 2024-03-04T16:34:58.323 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:58.323 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:34:58.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:58.332+0000 7fbbae00f800 0 bucket checkpoint complete 2024-03-04T16:34:58.342 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:34:58.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:58.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:34:58.343 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:34:58.399 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:34:58.399 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:34:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:34:58.460+0000 7f86dc03a800 1 waiting for incremental sync to catch up: 2024-03-04T16:34:58.451 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:34:58.451 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:03.461+0000 7f86dc03a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:03.452 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:35:03.452 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:08.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:08.462+0000 7f86dc03a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:08.453 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:35:08.453 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:13.464+0000 7f86dc03a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:13.455 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.28.6, , ] 2024-03-04T16:35:13.455 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:18.465+0000 7f86dc03a800 1 bucket sync caught up with source: 2024-03-04T16:35:18.456 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:18.456 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:18.465+0000 7f86dc03a800 0 bucket checkpoint complete 2024-03-04T16:35:18.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:35:18.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:18.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:18.481 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:35:18.539 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:18.540 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:18.603+0000 7ffa6181e800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:18.594 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:18.594 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:23.604+0000 7ffa6181e800 1 bucket sync caught up with source: 2024-03-04T16:35:23.595 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:23.595 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:23.604+0000 7ffa6181e800 0 bucket checkpoint complete 2024-03-04T16:35:23.617 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:35:23.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:23.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:23.617 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:35:23.675 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:23.675 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:23.737+0000 7f3ba628a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:23.728 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:23.728 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:28.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:28.739+0000 7f3ba628a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:28.729 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:28.729 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:33.740+0000 7f3ba628a800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:33.730 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:33.731 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:38.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:38.742+0000 7f3ba628a800 1 bucket sync caught up with source: 2024-03-04T16:35:38.732 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:38.732 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:38.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:38.742+0000 7f3ba628a800 0 bucket checkpoint complete 2024-03-04T16:35:38.756 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:35:38.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:38.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:38.757 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:35:38.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:38.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:38.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:38.880+0000 7fd9fae20800 1 bucket sync caught up with source: 2024-03-04T16:35:38.870 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:38.870 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:35:38.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:38.880+0000 7fd9fae20800 0 bucket checkpoint complete 2024-03-04T16:35:38.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:35:38.889 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.889 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.889 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.889 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:38.889 DEBUG:boto:Data: 2024-03-04T16:35:38.889 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.889 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.889 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.889 DEBUG:boto:Params: {} 2024-03-04T16:35:38.890 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:35:38.890 DEBUG:boto:Token: None 2024-03-04T16:35:38.891 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.891 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vo9jpTayo6YKOM1/PsQsQViXEm8= 2024-03-04T16:35:38.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vo9jpTayo6YKOM1/PsQsQViXEm8=', 'Content-Length': '0'} 2024-03-04T16:35:38.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002494497bf4785aa9-0065e5f85a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.896 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.896 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.897 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.897 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:38.897 DEBUG:boto:Data: 2024-03-04T16:35:38.897 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.897 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.897 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.897 DEBUG:boto:Params: {} 2024-03-04T16:35:38.897 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:35:38.897 DEBUG:boto:Token: None 2024-03-04T16:35:38.897 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.898 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vo9jpTayo6YKOM1/PsQsQViXEm8= 2024-03-04T16:35:38.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vo9jpTayo6YKOM1/PsQsQViXEm8=', 'Content-Length': '0'} 2024-03-04T16:35:38.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053ed6d80c1ac06f9-0065e5f85a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:35:38.903 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.903 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.903 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.903 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.903 DEBUG:boto:Method: GET 2024-03-04T16:35:38.903 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.903 DEBUG:boto:Data: 2024-03-04T16:35:38.903 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.903 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.903 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.903 DEBUG:boto:Params: {} 2024-03-04T16:35:38.904 DEBUG:boto:Token: None 2024-03-04T16:35:38.904 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/?versions 2024-03-04T16:35:38.904 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI= 2024-03-04T16:35:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI=', 'Content-Length': '0'} 2024-03-04T16:35:38.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043ad3e39b6ecb522-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.906 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:38.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:38.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:35:38.908 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.908 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.908 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.908 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.908 DEBUG:boto:Method: GET 2024-03-04T16:35:38.908 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.908 DEBUG:boto:Data: 2024-03-04T16:35:38.908 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.908 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.909 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.909 DEBUG:boto:Params: {} 2024-03-04T16:35:38.909 DEBUG:boto:Token: None 2024-03-04T16:35:38.909 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/?versions 2024-03-04T16:35:38.909 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI= 2024-03-04T16:35:38.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI=', 'Content-Length': '0'} 2024-03-04T16:35:38.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9849246f7306f99-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.912 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:38.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:38.912 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.912 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.913 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.913 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.913 DEBUG:boto:Method: GET 2024-03-04T16:35:38.913 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.913 DEBUG:boto:Data: 2024-03-04T16:35:38.913 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.913 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.913 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.913 DEBUG:boto:Params: {} 2024-03-04T16:35:38.913 DEBUG:boto:Token: None 2024-03-04T16:35:38.913 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/?versions 2024-03-04T16:35:38.913 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI= 2024-03-04T16:35:38.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI=', 'Content-Length': '0'} 2024-03-04T16:35:38.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a21cf095ab811b3-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.916 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.916 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.917 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.917 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.917 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.917 DEBUG:boto:Method: GET 2024-03-04T16:35:38.917 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:38.917 DEBUG:boto:Data: 2024-03-04T16:35:38.917 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.917 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.917 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.917 DEBUG:boto:Params: {} 2024-03-04T16:35:38.917 DEBUG:boto:Token: None 2024-03-04T16:35:38.918 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/?versions 2024-03-04T16:35:38.918 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI= 2024-03-04T16:35:38.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qtpXiZnj3tKZMJpwS04JDgkwRQI=', 'Content-Length': '0'} 2024-03-04T16:35:38.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07bf408d1b89c5a-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.920 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:38.921 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.921 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.921 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.921 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.921 DEBUG:boto:Method: GET 2024-03-04T16:35:38.921 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.921 DEBUG:boto:Data: 2024-03-04T16:35:38.921 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.922 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.922 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.922 DEBUG:boto:Params: {} 2024-03-04T16:35:38.922 DEBUG:boto:Token: None 2024-03-04T16:35:38.922 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.922 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e6460f406a34021-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.927 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.927 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.927 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.927 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.927 DEBUG:boto:Method: GET 2024-03-04T16:35:38.927 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.928 DEBUG:boto:Data: 2024-03-04T16:35:38.928 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.928 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.928 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.928 DEBUG:boto:Params: {} 2024-03-04T16:35:38.928 DEBUG:boto:Token: None 2024-03-04T16:35:38.928 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.928 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d60e40d82b50f423-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.931 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.931 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.931 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.931 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.931 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.931 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.931 DEBUG:boto:Data: 2024-03-04T16:35:38.931 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.932 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.932 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.932 DEBUG:boto:Params: {} 2024-03-04T16:35:38.932 DEBUG:boto:Token: None 2024-03-04T16:35:38.932 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.932 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vGOowYu0Folkp/DSuMwQK8hPttk= 2024-03-04T16:35:38.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vGOowYu0Folkp/DSuMwQK8hPttk=', 'Content-Length': '0'} 2024-03-04T16:35:38.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa7f1423700dcfd7-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.934 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.934 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.934 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.934 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.934 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.934 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.935 DEBUG:boto:Data: 2024-03-04T16:35:38.935 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.935 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.935 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.935 DEBUG:boto:Params: {} 2024-03-04T16:35:38.935 DEBUG:boto:Token: None 2024-03-04T16:35:38.935 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.935 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vGOowYu0Folkp/DSuMwQK8hPttk= 2024-03-04T16:35:38.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vGOowYu0Folkp/DSuMwQK8hPttk=', 'Content-Length': '0'} 2024-03-04T16:35:38.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5fddbf5c2f3e319-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:38.937 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.937 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.937 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.938 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.938 DEBUG:boto:Method: GET 2024-03-04T16:35:38.938 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.938 DEBUG:boto:Data: 2024-03-04T16:35:38.938 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.938 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.938 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.938 DEBUG:boto:Params: {} 2024-03-04T16:35:38.938 DEBUG:boto:Token: None 2024-03-04T16:35:38.938 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.938 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006575ffb235f5afc4-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.941 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.941 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.941 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.941 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.941 DEBUG:boto:Method: GET 2024-03-04T16:35:38.941 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.941 DEBUG:boto:Data: 2024-03-04T16:35:38.941 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.941 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.941 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.941 DEBUG:boto:Params: {} 2024-03-04T16:35:38.942 DEBUG:boto:Token: None 2024-03-04T16:35:38.942 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ecb06f0ab3d63d7-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.944 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.944 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.944 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.944 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.944 DEBUG:boto:Method: GET 2024-03-04T16:35:38.944 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:38.944 DEBUG:boto:Data: 2024-03-04T16:35:38.944 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.944 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.944 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.945 DEBUG:boto:Params: {} 2024-03-04T16:35:38.945 DEBUG:boto:Token: None 2024-03-04T16:35:38.945 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.945 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ= 2024-03-04T16:35:38.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057ffe13ce03ab37e-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.947 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.948 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.948 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.948 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.948 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.948 DEBUG:boto:Method: GET 2024-03-04T16:35:38.948 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:38.948 DEBUG:boto:Data: 2024-03-04T16:35:38.948 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.948 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.948 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.949 DEBUG:boto:Params: {} 2024-03-04T16:35:38.949 DEBUG:boto:Token: None 2024-03-04T16:35:38.949 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.949 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ= 2024-03-04T16:35:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeceab74c97d2f51-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.951 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.952 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.952 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.952 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.952 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:38.952 DEBUG:boto:Data: 2024-03-04T16:35:38.952 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.952 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.952 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.952 DEBUG:boto:Params: {} 2024-03-04T16:35:38.953 DEBUG:boto:Token: None 2024-03-04T16:35:38.953 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj 2024-03-04T16:35:38.953 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IWFPPycOW/LVS51SXJv4gUO0MlQ= 2024-03-04T16:35:38.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IWFPPycOW/LVS51SXJv4gUO0MlQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa01b01b819db605-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.955 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.955 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.955 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.955 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:38.956 DEBUG:boto:Data: 2024-03-04T16:35:38.956 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.956 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.956 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.956 DEBUG:boto:Params: {} 2024-03-04T16:35:38.956 DEBUG:boto:Token: None 2024-03-04T16:35:38.956 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj 2024-03-04T16:35:38.956 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IWFPPycOW/LVS51SXJv4gUO0MlQ= 2024-03-04T16:35:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IWFPPycOW/LVS51SXJv4gUO0MlQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000333d48b4abec1987-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:38.959 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.959 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.959 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.959 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.959 DEBUG:boto:Method: GET 2024-03-04T16:35:38.959 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.959 DEBUG:boto:Data: 2024-03-04T16:35:38.959 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.960 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.960 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.960 DEBUG:boto:Params: {} 2024-03-04T16:35:38.960 DEBUG:boto:Token: None 2024-03-04T16:35:38.960 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.960 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a24ddef9c39ed44a-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.962 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.962 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.962 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.962 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.963 DEBUG:boto:Method: GET 2024-03-04T16:35:38.963 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.963 DEBUG:boto:Data: 2024-03-04T16:35:38.963 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.963 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.963 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.963 DEBUG:boto:Params: {} 2024-03-04T16:35:38.963 DEBUG:boto:Token: None 2024-03-04T16:35:38.963 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:38.963 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8= 2024-03-04T16:35:38.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZwXagdzQ88v/O3bktXcqEHhXrC8=', 'Content-Length': '0'} 2024-03-04T16:35:38.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a02a15e3d560b16-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.965 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.966 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.966 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.966 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.966 DEBUG:boto:Method: GET 2024-03-04T16:35:38.966 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:38.966 DEBUG:boto:Data: 2024-03-04T16:35:38.966 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.966 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.966 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.966 DEBUG:boto:Params: {} 2024-03-04T16:35:38.966 DEBUG:boto:Token: None 2024-03-04T16:35:38.966 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.967 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ= 2024-03-04T16:35:38.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d7f182fb032821-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.969 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.969 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.970 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.970 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.970 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.970 DEBUG:boto:Method: GET 2024-03-04T16:35:38.970 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:38.970 DEBUG:boto:Data: 2024-03-04T16:35:38.970 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.970 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.970 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.970 DEBUG:boto:Params: {} 2024-03-04T16:35:38.970 DEBUG:boto:Token: None 2024-03-04T16:35:38.970 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.971 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ= 2024-03-04T16:35:38.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c66be260939119-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.973 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:38.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:38.974 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.974 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.974 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.974 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.974 DEBUG:boto:Method: GET 2024-03-04T16:35:38.974 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.974 DEBUG:boto:Data: 2024-03-04T16:35:38.974 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.974 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.974 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.974 DEBUG:boto:Params: {} 2024-03-04T16:35:38.975 DEBUG:boto:Token: None 2024-03-04T16:35:38.975 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.975 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs= 2024-03-04T16:35:38.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs=', 'Content-Length': '0'} 2024-03-04T16:35:38.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000baf906db7a811aec-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.976 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.977 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.977 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.977 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.977 DEBUG:boto:Method: GET 2024-03-04T16:35:38.977 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.977 DEBUG:boto:Data: 2024-03-04T16:35:38.977 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.977 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.977 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.977 DEBUG:boto:Params: {} 2024-03-04T16:35:38.977 DEBUG:boto:Token: None 2024-03-04T16:35:38.978 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.978 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs= 2024-03-04T16:35:38.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs=', 'Content-Length': '0'} 2024-03-04T16:35:38.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052239638f429f253-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.980 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.980 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.980 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.980 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.980 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.980 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.980 DEBUG:boto:Data: 2024-03-04T16:35:38.980 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.981 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.981 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.981 DEBUG:boto:Params: {} 2024-03-04T16:35:38.981 DEBUG:boto:Token: None 2024-03-04T16:35:38.981 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.981 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oQ5jcoUhXUb9l8Y+i2fOOiyduSU= 2024-03-04T16:35:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oQ5jcoUhXUb9l8Y+i2fOOiyduSU=', 'Content-Length': '0'} 2024-03-04T16:35:38.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006752791d11d28b9b-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.983 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.983 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.983 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.983 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.983 DEBUG:boto:Method: HEAD 2024-03-04T16:35:38.983 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.983 DEBUG:boto:Data: 2024-03-04T16:35:38.983 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.984 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.984 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.984 DEBUG:boto:Params: {} 2024-03-04T16:35:38.984 DEBUG:boto:Token: None 2024-03-04T16:35:38.984 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.984 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oQ5jcoUhXUb9l8Y+i2fOOiyduSU= 2024-03-04T16:35:38.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oQ5jcoUhXUb9l8Y+i2fOOiyduSU=', 'Content-Length': '0'} 2024-03-04T16:35:38.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000425e5f0df9eceeb8-0065e5f85a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:38.986 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.986 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.986 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.986 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.986 DEBUG:boto:Method: GET 2024-03-04T16:35:38.987 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.987 DEBUG:boto:Data: 2024-03-04T16:35:38.987 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.987 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.987 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.987 DEBUG:boto:Params: {} 2024-03-04T16:35:38.987 DEBUG:boto:Token: None 2024-03-04T16:35:38.987 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.987 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs= 2024-03-04T16:35:38.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs=', 'Content-Length': '0'} 2024-03-04T16:35:38.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f45c98ac04e7f281-0065e5f85a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:38 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.989 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:38.989 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:38.989 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.989 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.990 DEBUG:boto:Method: GET 2024-03-04T16:35:38.990 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.990 DEBUG:boto:Data: 2024-03-04T16:35:38.990 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.990 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:38.990 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.990 DEBUG:boto:Params: {} 2024-03-04T16:35:38.990 DEBUG:boto:Token: None 2024-03-04T16:35:38.990 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:38.990 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs= 2024-03-04T16:35:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rH6KzBsV3KEPVt2inQCbQaWKTRs=', 'Content-Length': '0'} 2024-03-04T16:35:38.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db3ac9f6745af61d-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:38.992 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:38.992 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:38.992 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.992 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:38.993 DEBUG:boto:Method: GET 2024-03-04T16:35:38.993 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:38.993 DEBUG:boto:Data: 2024-03-04T16:35:38.993 DEBUG:boto:Headers: {} 2024-03-04T16:35:38.993 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:38.993 DEBUG:boto:Port: 8000 2024-03-04T16:35:38.993 DEBUG:boto:Params: {} 2024-03-04T16:35:38.993 DEBUG:boto:Token: None 2024-03-04T16:35:38.993 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:38 GMT /sbnvju-23/ 2024-03-04T16:35:38.993 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ= 2024-03-04T16:35:38.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:38 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LfawIBc2cPh0LfnDiePyuCqO6KQ=', 'Content-Length': '0'} 2024-03-04T16:35:38.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b23ff9bac34ba515-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.036 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.036 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.036 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.037 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.037 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.037 DEBUG:boto:Method: GET 2024-03-04T16:35:39.037 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.037 DEBUG:boto:Data: 2024-03-04T16:35:39.037 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.037 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.037 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.037 DEBUG:boto:Params: {} 2024-03-04T16:35:39.037 DEBUG:boto:Token: None 2024-03-04T16:35:39.037 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.038 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c19d2e656893c9-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.041 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.041 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.041 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.041 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.041 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.041 DEBUG:boto:Data: 2024-03-04T16:35:39.041 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.041 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.041 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.042 DEBUG:boto:Params: {} 2024-03-04T16:35:39.042 DEBUG:boto:Token: None 2024-03-04T16:35:39.042 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.042 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006dea09be88a89ca-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.044 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.045 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.045 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.045 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.045 DEBUG:boto:Data: 2024-03-04T16:35:39.045 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.045 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.045 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.045 DEBUG:boto:Params: {} 2024-03-04T16:35:39.045 DEBUG:boto:Token: None 2024-03-04T16:35:39.045 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.045 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ff237148e34c206-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.048 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.048 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.048 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.048 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.048 DEBUG:boto:Method: GET 2024-03-04T16:35:39.049 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.049 DEBUG:boto:Data: 2024-03-04T16:35:39.049 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.049 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.049 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.049 DEBUG:boto:Params: {} 2024-03-04T16:35:39.049 DEBUG:boto:Token: None 2024-03-04T16:35:39.049 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8b01d5e01f261cd-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.051 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.051 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.052 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.052 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.052 DEBUG:boto:Method: GET 2024-03-04T16:35:39.052 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.052 DEBUG:boto:Data: 2024-03-04T16:35:39.052 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.052 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.052 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.052 DEBUG:boto:Params: {} 2024-03-04T16:35:39.052 DEBUG:boto:Token: None 2024-03-04T16:35:39.053 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d63aaca473518148-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.055 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.055 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.055 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.055 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.055 DEBUG:boto:Method: GET 2024-03-04T16:35:39.055 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.055 DEBUG:boto:Data: 2024-03-04T16:35:39.055 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.055 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.056 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.056 DEBUG:boto:Params: {} 2024-03-04T16:35:39.056 DEBUG:boto:Token: None 2024-03-04T16:35:39.056 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.056 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc99a5ac644961a-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.058 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.059 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.059 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.059 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.059 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.059 DEBUG:boto:Method: GET 2024-03-04T16:35:39.059 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.059 DEBUG:boto:Data: 2024-03-04T16:35:39.059 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.059 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.060 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.060 DEBUG:boto:Params: {} 2024-03-04T16:35:39.060 DEBUG:boto:Token: None 2024-03-04T16:35:39.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.060 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e1b2c133b6a2fac-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.063 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.063 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:35:39.063 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:35:39.063 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.063 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.064 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.064 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:39.064 DEBUG:boto:Data: 2024-03-04T16:35:39.064 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.064 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.064 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.064 DEBUG:boto:Params: {} 2024-03-04T16:35:39.064 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:35:39.064 DEBUG:boto:Token: None 2024-03-04T16:35:39.065 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc= 2024-03-04T16:35:39.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc=', 'Content-Length': '0'} 2024-03-04T16:35:39.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ebe05193f8afe0f-0065e5f85b-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.069 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.069 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.069 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.069 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:39.069 DEBUG:boto:Data: 2024-03-04T16:35:39.069 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.070 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.070 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.070 DEBUG:boto:Params: {} 2024-03-04T16:35:39.070 DEBUG:boto:Token: None 2024-03-04T16:35:39.070 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.070 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc= 2024-03-04T16:35:39.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc=', 'Content-Length': '0'} 2024-03-04T16:35:39.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000245c6f96cfd2ed98-0065e5f85b-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:35:39.072 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.072 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.073 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.073 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.073 DEBUG:boto:Method: GET 2024-03-04T16:35:39.073 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.073 DEBUG:boto:Data: 2024-03-04T16:35:39.073 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.073 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.073 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.073 DEBUG:boto:Params: {} 2024-03-04T16:35:39.073 DEBUG:boto:Token: None 2024-03-04T16:35:39.073 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.074 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c008c081fe5956-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.076 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.077 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:35:39.077 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.077 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.077 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.077 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.077 DEBUG:boto:Method: GET 2024-03-04T16:35:39.077 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.077 DEBUG:boto:Data: 2024-03-04T16:35:39.078 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.078 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.078 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.078 DEBUG:boto:Params: {} 2024-03-04T16:35:39.078 DEBUG:boto:Token: None 2024-03-04T16:35:39.078 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.078 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d0748667e49067-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.081 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.081 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.081 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.082 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.082 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.082 DEBUG:boto:Method: GET 2024-03-04T16:35:39.082 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.082 DEBUG:boto:Data: 2024-03-04T16:35:39.082 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.082 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.082 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.082 DEBUG:boto:Params: {} 2024-03-04T16:35:39.082 DEBUG:boto:Token: None 2024-03-04T16:35:39.082 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.083 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9b77a5cf2aab7c0-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.085 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.085 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.086 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.086 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.086 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.086 DEBUG:boto:Method: GET 2024-03-04T16:35:39.086 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.086 DEBUG:boto:Data: 2024-03-04T16:35:39.086 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.086 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.086 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.086 DEBUG:boto:Params: {} 2024-03-04T16:35:39.087 DEBUG:boto:Token: None 2024-03-04T16:35:39.087 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.087 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120d1f11eeb1855e-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.132 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.133 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.133 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.133 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.133 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.133 DEBUG:boto:Method: GET 2024-03-04T16:35:39.133 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.133 DEBUG:boto:Data: 2024-03-04T16:35:39.133 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.133 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.133 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.133 DEBUG:boto:Params: {} 2024-03-04T16:35:39.134 DEBUG:boto:Token: None 2024-03-04T16:35:39.134 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.134 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092970975a9823e5e-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.136 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.136 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.136 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.136 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.136 DEBUG:boto:Method: GET 2024-03-04T16:35:39.136 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.137 DEBUG:boto:Data: 2024-03-04T16:35:39.137 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.137 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.137 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.137 DEBUG:boto:Params: {} 2024-03-04T16:35:39.137 DEBUG:boto:Token: None 2024-03-04T16:35:39.137 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.137 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000988088b6db1f91de-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.140 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.140 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.140 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.140 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.140 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.140 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.140 DEBUG:boto:Data: 2024-03-04T16:35:39.140 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.140 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.140 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.141 DEBUG:boto:Params: {} 2024-03-04T16:35:39.141 DEBUG:boto:Token: None 2024-03-04T16:35:39.141 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.141 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE= 2024-03-04T16:35:39.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE=', 'Content-Length': '0'} 2024-03-04T16:35:39.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab313c962e3f0225-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.143 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.143 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.143 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.144 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.144 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.144 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.144 DEBUG:boto:Data: 2024-03-04T16:35:39.144 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.144 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.144 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.144 DEBUG:boto:Params: {} 2024-03-04T16:35:39.144 DEBUG:boto:Token: None 2024-03-04T16:35:39.144 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE= 2024-03-04T16:35:39.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE=', 'Content-Length': '0'} 2024-03-04T16:35:39.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f40547043ce1a13d-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.147 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.147 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.147 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.147 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.147 DEBUG:boto:Method: GET 2024-03-04T16:35:39.147 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.147 DEBUG:boto:Data: 2024-03-04T16:35:39.147 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.147 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.147 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.147 DEBUG:boto:Params: {} 2024-03-04T16:35:39.148 DEBUG:boto:Token: None 2024-03-04T16:35:39.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.148 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d6483e7068d90a4e-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.150 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.150 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.150 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.150 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.150 DEBUG:boto:Method: GET 2024-03-04T16:35:39.150 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.150 DEBUG:boto:Data: 2024-03-04T16:35:39.150 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.150 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.151 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.151 DEBUG:boto:Params: {} 2024-03-04T16:35:39.151 DEBUG:boto:Token: None 2024-03-04T16:35:39.151 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.151 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012b4a1c056fc97fb-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.153 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.153 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.153 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.154 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.154 DEBUG:boto:Method: GET 2024-03-04T16:35:39.154 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.154 DEBUG:boto:Data: 2024-03-04T16:35:39.154 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.154 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.154 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.154 DEBUG:boto:Params: {} 2024-03-04T16:35:39.154 DEBUG:boto:Token: None 2024-03-04T16:35:39.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039903f657424022e-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.157 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.157 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.157 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.158 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.158 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.158 DEBUG:boto:Method: GET 2024-03-04T16:35:39.158 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.158 DEBUG:boto:Data: 2024-03-04T16:35:39.158 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.158 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.158 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.158 DEBUG:boto:Params: {} 2024-03-04T16:35:39.158 DEBUG:boto:Token: None 2024-03-04T16:35:39.159 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aedb5980ca3f544-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.161 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.162 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.162 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.162 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.162 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.162 DEBUG:boto:Data: 2024-03-04T16:35:39.162 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.162 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.162 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.162 DEBUG:boto:Params: {} 2024-03-04T16:35:39.162 DEBUG:boto:Token: None 2024-03-04T16:35:39.163 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.163 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e24bca06324f80d-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.165 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.165 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.165 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.165 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.165 DEBUG:boto:Data: 2024-03-04T16:35:39.165 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.166 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.166 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.166 DEBUG:boto:Params: {} 2024-03-04T16:35:39.166 DEBUG:boto:Token: None 2024-03-04T16:35:39.166 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.166 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007303ca60afa99f9c-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.169 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.169 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.169 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.169 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.169 DEBUG:boto:Method: GET 2024-03-04T16:35:39.169 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.169 DEBUG:boto:Data: 2024-03-04T16:35:39.169 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.169 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.169 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.169 DEBUG:boto:Params: {} 2024-03-04T16:35:39.169 DEBUG:boto:Token: None 2024-03-04T16:35:39.170 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.170 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000684f70b5cda52833-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.172 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.172 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.172 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.172 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.172 DEBUG:boto:Method: GET 2024-03-04T16:35:39.172 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.172 DEBUG:boto:Data: 2024-03-04T16:35:39.172 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.173 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.173 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.173 DEBUG:boto:Params: {} 2024-03-04T16:35:39.173 DEBUG:boto:Token: None 2024-03-04T16:35:39.173 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.173 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad64893611490aa7-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.175 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.175 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.175 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.175 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.175 DEBUG:boto:Method: GET 2024-03-04T16:35:39.176 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.176 DEBUG:boto:Data: 2024-03-04T16:35:39.176 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.176 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.176 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.176 DEBUG:boto:Params: {} 2024-03-04T16:35:39.176 DEBUG:boto:Token: None 2024-03-04T16:35:39.176 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.176 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8015082bc97cd55-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.220 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.220 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.220 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.221 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.221 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.221 DEBUG:boto:Method: GET 2024-03-04T16:35:39.221 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.221 DEBUG:boto:Data: 2024-03-04T16:35:39.221 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.221 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.221 DEBUG:boto:Params: {} 2024-03-04T16:35:39.221 DEBUG:boto:Token: None 2024-03-04T16:35:39.221 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099db29981fa578f1-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.224 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.225 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.225 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.225 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.225 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.225 DEBUG:boto:Method: GET 2024-03-04T16:35:39.225 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.225 DEBUG:boto:Data: 2024-03-04T16:35:39.225 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.226 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.226 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.226 DEBUG:boto:Params: {} 2024-03-04T16:35:39.226 DEBUG:boto:Token: None 2024-03-04T16:35:39.226 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.226 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab0ddc2c89af2702-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.228 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.228 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.228 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.228 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.228 DEBUG:boto:Method: GET 2024-03-04T16:35:39.229 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.229 DEBUG:boto:Data: 2024-03-04T16:35:39.229 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.229 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.229 DEBUG:boto:Params: {} 2024-03-04T16:35:39.229 DEBUG:boto:Token: None 2024-03-04T16:35:39.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6bf77216d51f089-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.231 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.232 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.232 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.232 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.232 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.232 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.232 DEBUG:boto:Data: 2024-03-04T16:35:39.232 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.232 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.232 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.232 DEBUG:boto:Params: {} 2024-03-04T16:35:39.233 DEBUG:boto:Token: None 2024-03-04T16:35:39.233 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.233 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s= 2024-03-04T16:35:39.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s=', 'Content-Length': '0'} 2024-03-04T16:35:39.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035bbe92830a1d99f-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.235 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.235 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.235 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.235 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.235 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.235 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.236 DEBUG:boto:Data: 2024-03-04T16:35:39.236 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.236 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.236 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.236 DEBUG:boto:Params: {} 2024-03-04T16:35:39.236 DEBUG:boto:Token: None 2024-03-04T16:35:39.236 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.236 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s= 2024-03-04T16:35:39.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s=', 'Content-Length': '0'} 2024-03-04T16:35:39.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081692c603a1e4408-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.238 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.238 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.239 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.239 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.239 DEBUG:boto:Method: GET 2024-03-04T16:35:39.239 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.239 DEBUG:boto:Data: 2024-03-04T16:35:39.239 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.239 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.239 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.239 DEBUG:boto:Params: {} 2024-03-04T16:35:39.239 DEBUG:boto:Token: None 2024-03-04T16:35:39.239 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.240 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f739bae20166263-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.241 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.242 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.242 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.242 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.242 DEBUG:boto:Method: GET 2024-03-04T16:35:39.242 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.242 DEBUG:boto:Data: 2024-03-04T16:35:39.242 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.242 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.242 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.242 DEBUG:boto:Params: {} 2024-03-04T16:35:39.242 DEBUG:boto:Token: None 2024-03-04T16:35:39.243 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.243 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bf4d0b77b93f285-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.245 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.245 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.245 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.245 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.245 DEBUG:boto:Method: GET 2024-03-04T16:35:39.245 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.245 DEBUG:boto:Data: 2024-03-04T16:35:39.245 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.245 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.246 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.246 DEBUG:boto:Params: {} 2024-03-04T16:35:39.246 DEBUG:boto:Token: None 2024-03-04T16:35:39.246 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.246 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef73fe6112b6f2ca-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.249 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.249 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.249 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.249 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.249 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.249 DEBUG:boto:Method: GET 2024-03-04T16:35:39.249 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.250 DEBUG:boto:Data: 2024-03-04T16:35:39.250 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.250 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.250 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.250 DEBUG:boto:Params: {} 2024-03-04T16:35:39.250 DEBUG:boto:Token: None 2024-03-04T16:35:39.250 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.250 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8d5c99793f83b32-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.253 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.253 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.253 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.253 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.253 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.253 DEBUG:boto:Data: 2024-03-04T16:35:39.253 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.254 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.254 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.254 DEBUG:boto:Params: {} 2024-03-04T16:35:39.254 DEBUG:boto:Token: None 2024-03-04T16:35:39.254 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.254 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca7ced64f59aae62-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.256 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.257 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.257 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.257 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.257 DEBUG:boto:Data: 2024-03-04T16:35:39.257 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.257 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.257 DEBUG:boto:Params: {} 2024-03-04T16:35:39.257 DEBUG:boto:Token: None 2024-03-04T16:35:39.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050db7f33b18b78eb-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.260 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.260 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.260 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.260 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.260 DEBUG:boto:Method: GET 2024-03-04T16:35:39.260 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.261 DEBUG:boto:Data: 2024-03-04T16:35:39.261 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.261 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.261 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.261 DEBUG:boto:Params: {} 2024-03-04T16:35:39.261 DEBUG:boto:Token: None 2024-03-04T16:35:39.261 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.261 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e83d26716c5cf1a9-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.263 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.263 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.263 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.263 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.263 DEBUG:boto:Method: GET 2024-03-04T16:35:39.263 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.264 DEBUG:boto:Data: 2024-03-04T16:35:39.264 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.264 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.264 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.264 DEBUG:boto:Params: {} 2024-03-04T16:35:39.264 DEBUG:boto:Token: None 2024-03-04T16:35:39.264 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.264 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9f65e04d3578efa-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.267 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.267 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.267 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.267 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.267 DEBUG:boto:Method: GET 2024-03-04T16:35:39.267 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.267 DEBUG:boto:Data: 2024-03-04T16:35:39.267 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.267 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.267 DEBUG:boto:Params: {} 2024-03-04T16:35:39.268 DEBUG:boto:Token: None 2024-03-04T16:35:39.268 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.268 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d190fef1c673dc3-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.312 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.313 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.313 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.313 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.313 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.313 DEBUG:boto:Method: GET 2024-03-04T16:35:39.313 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.313 DEBUG:boto:Data: 2024-03-04T16:35:39.313 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.313 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.313 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.313 DEBUG:boto:Params: {} 2024-03-04T16:35:39.313 DEBUG:boto:Token: None 2024-03-04T16:35:39.314 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.314 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078676204b50ca831-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.317 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.317 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:35:39.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:35:39.317 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.317 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.317 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.318 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:39.318 DEBUG:boto:Data: 2024-03-04T16:35:39.318 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.318 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.318 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.318 DEBUG:boto:Params: {} 2024-03-04T16:35:39.318 DEBUG:boto:Token: None 2024-03-04T16:35:39.318 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.318 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc= 2024-03-04T16:35:39.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc=', 'Content-Length': '0'} 2024-03-04T16:35:39.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d83308d29c5d7eb-0065e5f85b-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.321 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.321 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.321 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.321 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:39.321 DEBUG:boto:Data: 2024-03-04T16:35:39.321 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.321 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.322 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.322 DEBUG:boto:Params: {} 2024-03-04T16:35:39.322 DEBUG:boto:Token: None 2024-03-04T16:35:39.322 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.322 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc= 2024-03-04T16:35:39.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:P75ieKdL3Re0FgEqyqrGik9e/uc=', 'Content-Length': '0'} 2024-03-04T16:35:39.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7146c45df52edcf-0065e5f85b-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.325 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:35:39.325 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.325 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.325 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.325 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.325 DEBUG:boto:Method: GET 2024-03-04T16:35:39.325 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.325 DEBUG:boto:Data: 2024-03-04T16:35:39.325 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.325 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.326 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.326 DEBUG:boto:Params: {} 2024-03-04T16:35:39.326 DEBUG:boto:Token: None 2024-03-04T16:35:39.326 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8957ec71b32876-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.329 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.329 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:35:39.329 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.329 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.329 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.330 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.330 DEBUG:boto:Method: GET 2024-03-04T16:35:39.330 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.330 DEBUG:boto:Data: 2024-03-04T16:35:39.330 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.330 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.330 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.330 DEBUG:boto:Params: {} 2024-03-04T16:35:39.330 DEBUG:boto:Token: None 2024-03-04T16:35:39.330 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.330 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cca7e6cb13c1ac1-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.376 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:35:39.377 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.377 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.377 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.377 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.377 DEBUG:boto:Method: GET 2024-03-04T16:35:39.377 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.377 DEBUG:boto:Data: 2024-03-04T16:35:39.377 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.377 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.377 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.377 DEBUG:boto:Params: {} 2024-03-04T16:35:39.378 DEBUG:boto:Token: None 2024-03-04T16:35:39.378 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.378 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a47163358f24ce-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.424 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.424 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.424 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.424 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.425 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.425 DEBUG:boto:Method: GET 2024-03-04T16:35:39.425 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:35:39.425 DEBUG:boto:Data: 2024-03-04T16:35:39.425 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.425 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.425 DEBUG:boto:Params: {} 2024-03-04T16:35:39.425 DEBUG:boto:Token: None 2024-03-04T16:35:39.425 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versions 2024-03-04T16:35:39.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM= 2024-03-04T16:35:39.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:suxHUkKGi01XpZRlYJgPMIoxFhM=', 'Content-Length': '0'} 2024-03-04T16:35:39.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000217f66c060689927-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.429 DEBUG:boto:b'sbnvju-23999falseobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1true2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-03-04T16:34:06.406Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.429 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.429 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.429 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.429 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.429 DEBUG:boto:Method: GET 2024-03-04T16:35:39.429 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.429 DEBUG:boto:Data: 2024-03-04T16:35:39.430 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.430 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.430 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.430 DEBUG:boto:Params: {} 2024-03-04T16:35:39.430 DEBUG:boto:Token: None 2024-03-04T16:35:39.430 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a2b7712ce818e2c-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.432 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.432 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.432 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.432 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.432 DEBUG:boto:Method: GET 2024-03-04T16:35:39.433 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.433 DEBUG:boto:Data: 2024-03-04T16:35:39.433 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.433 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.433 DEBUG:boto:Params: {} 2024-03-04T16:35:39.433 DEBUG:boto:Token: None 2024-03-04T16:35:39.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046be22ffbee02b83-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.436 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.436 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.436 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.436 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.436 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.436 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.436 DEBUG:boto:Data: 2024-03-04T16:35:39.436 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.436 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.437 DEBUG:boto:Params: {} 2024-03-04T16:35:39.437 DEBUG:boto:Token: None 2024-03-04T16:35:39.437 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE= 2024-03-04T16:35:39.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE=', 'Content-Length': '0'} 2024-03-04T16:35:39.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004230a9f8890c95d2-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.439 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.439 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.439 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.439 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.439 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.439 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.439 DEBUG:boto:Data: 2024-03-04T16:35:39.439 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.439 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.439 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.439 DEBUG:boto:Params: {} 2024-03-04T16:35:39.440 DEBUG:boto:Token: None 2024-03-04T16:35:39.440 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.440 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE= 2024-03-04T16:35:39.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ituJW9wUeuzduYmrU5soUtp6PzE=', 'Content-Length': '0'} 2024-03-04T16:35:39.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000324ddd03e29ba60d-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.442 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.442 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.442 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.442 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.442 DEBUG:boto:Method: GET 2024-03-04T16:35:39.443 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.443 DEBUG:boto:Data: 2024-03-04T16:35:39.443 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.443 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.443 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.443 DEBUG:boto:Params: {} 2024-03-04T16:35:39.443 DEBUG:boto:Token: None 2024-03-04T16:35:39.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ccf287c959b7366-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.445 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.445 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.445 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.446 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.446 DEBUG:boto:Method: GET 2024-03-04T16:35:39.446 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.446 DEBUG:boto:Data: 2024-03-04T16:35:39.446 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.446 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.446 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.446 DEBUG:boto:Params: {} 2024-03-04T16:35:39.446 DEBUG:boto:Token: None 2024-03-04T16:35:39.446 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.446 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2e90d3199217d39-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.448 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.449 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.449 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.449 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.449 DEBUG:boto:Method: GET 2024-03-04T16:35:39.449 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.449 DEBUG:boto:Data: 2024-03-04T16:35:39.449 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.449 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.449 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.449 DEBUG:boto:Params: {} 2024-03-04T16:35:39.449 DEBUG:boto:Token: None 2024-03-04T16:35:39.449 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.450 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072cb136219a07e51-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.452 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.453 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.453 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.453 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.453 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.453 DEBUG:boto:Method: GET 2024-03-04T16:35:39.453 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.453 DEBUG:boto:Data: 2024-03-04T16:35:39.453 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.453 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.453 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.453 DEBUG:boto:Params: {} 2024-03-04T16:35:39.453 DEBUG:boto:Token: None 2024-03-04T16:35:39.454 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.454 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b5ad507c34955a-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.456 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.457 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.457 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.457 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.457 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.457 DEBUG:boto:Data: 2024-03-04T16:35:39.457 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.457 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.457 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.457 DEBUG:boto:Params: {} 2024-03-04T16:35:39.458 DEBUG:boto:Token: None 2024-03-04T16:35:39.458 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.458 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000416330282851ad3a-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.460 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.461 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.461 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.461 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.461 DEBUG:boto:Data: 2024-03-04T16:35:39.461 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.461 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.461 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.461 DEBUG:boto:Params: {} 2024-03-04T16:35:39.461 DEBUG:boto:Token: None 2024-03-04T16:35:39.461 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.461 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000988a3aa7d3fb5492-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.464 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.464 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.464 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.464 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.464 DEBUG:boto:Method: GET 2024-03-04T16:35:39.464 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.464 DEBUG:boto:Data: 2024-03-04T16:35:39.464 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.465 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.465 DEBUG:boto:Params: {} 2024-03-04T16:35:39.465 DEBUG:boto:Token: None 2024-03-04T16:35:39.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a40abbd2e96070b-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.467 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.467 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.467 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.467 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.467 DEBUG:boto:Method: GET 2024-03-04T16:35:39.467 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.467 DEBUG:boto:Data: 2024-03-04T16:35:39.468 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.468 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.468 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.468 DEBUG:boto:Params: {} 2024-03-04T16:35:39.468 DEBUG:boto:Token: None 2024-03-04T16:35:39.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049bf9a275d461829-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.470 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.470 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.470 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.470 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.471 DEBUG:boto:Method: GET 2024-03-04T16:35:39.471 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.471 DEBUG:boto:Data: 2024-03-04T16:35:39.471 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.471 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.471 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.471 DEBUG:boto:Params: {} 2024-03-04T16:35:39.471 DEBUG:boto:Token: None 2024-03-04T16:35:39.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.471 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808c07a6d066b906-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.516 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.516 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.516 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.517 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.517 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.517 DEBUG:boto:Method: GET 2024-03-04T16:35:39.517 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.517 DEBUG:boto:Data: 2024-03-04T16:35:39.517 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.517 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.517 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.517 DEBUG:boto:Params: {} 2024-03-04T16:35:39.517 DEBUG:boto:Token: None 2024-03-04T16:35:39.517 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006391276a8f2292a8-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.520 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.521 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.521 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.521 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.521 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.521 DEBUG:boto:Method: GET 2024-03-04T16:35:39.521 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.521 DEBUG:boto:Data: 2024-03-04T16:35:39.522 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.522 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.522 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.522 DEBUG:boto:Params: {} 2024-03-04T16:35:39.522 DEBUG:boto:Token: None 2024-03-04T16:35:39.522 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.522 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d1be2fd8f39b20f-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.524 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.524 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.524 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.524 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.524 DEBUG:boto:Method: GET 2024-03-04T16:35:39.525 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.525 DEBUG:boto:Data: 2024-03-04T16:35:39.525 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.525 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.525 DEBUG:boto:Params: {} 2024-03-04T16:35:39.525 DEBUG:boto:Token: None 2024-03-04T16:35:39.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b954c356f278c770-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.527 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.527 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.528 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.528 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.528 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.528 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.528 DEBUG:boto:Data: 2024-03-04T16:35:39.528 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.528 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.528 DEBUG:boto:Params: {} 2024-03-04T16:35:39.528 DEBUG:boto:Token: None 2024-03-04T16:35:39.528 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.529 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s= 2024-03-04T16:35:39.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s=', 'Content-Length': '0'} 2024-03-04T16:35:39.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000164f429481fb94f5-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.530 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.530 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.531 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.531 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.531 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.531 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.531 DEBUG:boto:Data: 2024-03-04T16:35:39.531 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.531 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.531 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.531 DEBUG:boto:Params: {} 2024-03-04T16:35:39.531 DEBUG:boto:Token: None 2024-03-04T16:35:39.531 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.532 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s= 2024-03-04T16:35:39.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:fum9EOGlp9q/srHQLFpmIHDsq9s=', 'Content-Length': '0'} 2024-03-04T16:35:39.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed2284ff360cb415-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.534 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.534 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.534 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.534 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.534 DEBUG:boto:Method: GET 2024-03-04T16:35:39.534 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.534 DEBUG:boto:Data: 2024-03-04T16:35:39.534 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.534 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.534 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.535 DEBUG:boto:Params: {} 2024-03-04T16:35:39.535 DEBUG:boto:Token: None 2024-03-04T16:35:39.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.535 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008969ddefe52e4fde-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.537 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.537 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.537 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.537 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.537 DEBUG:boto:Method: GET 2024-03-04T16:35:39.537 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.537 DEBUG:boto:Data: 2024-03-04T16:35:39.537 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.537 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.538 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.538 DEBUG:boto:Params: {} 2024-03-04T16:35:39.538 DEBUG:boto:Token: None 2024-03-04T16:35:39.538 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:35:39.538 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU= 2024-03-04T16:35:39.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DXfOv4BLMVwtKop3GBzb0UgjRRU=', 'Content-Length': '0'} 2024-03-04T16:35:39.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c90ba37d121e1bb-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.540 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.540 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.543 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.543 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.543 DEBUG:boto:Method: GET 2024-03-04T16:35:39.543 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.543 DEBUG:boto:Data: 2024-03-04T16:35:39.543 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.543 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.543 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.544 DEBUG:boto:Params: {} 2024-03-04T16:35:39.544 DEBUG:boto:Token: None 2024-03-04T16:35:39.544 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.544 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240f166da4d72c6f-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.547 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.547 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.547 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.547 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.547 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.547 DEBUG:boto:Method: GET 2024-03-04T16:35:39.547 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.548 DEBUG:boto:Data: 2024-03-04T16:35:39.548 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.548 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.548 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.548 DEBUG:boto:Params: {} 2024-03-04T16:35:39.548 DEBUG:boto:Token: None 2024-03-04T16:35:39.548 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d1bed3a14a1dfd-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.551 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.551 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.551 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.552 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.552 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.552 DEBUG:boto:Data: 2024-03-04T16:35:39.552 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.552 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.552 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.552 DEBUG:boto:Params: {} 2024-03-04T16:35:39.552 DEBUG:boto:Token: None 2024-03-04T16:35:39.552 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a874c4c472742848-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.555 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.555 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.555 DEBUG:boto:Method: HEAD 2024-03-04T16:35:39.555 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:39.555 DEBUG:boto:Data: 2024-03-04T16:35:39.555 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.555 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.555 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.556 DEBUG:boto:Params: {} 2024-03-04T16:35:39.556 DEBUG:boto:Token: None 2024-03-04T16:35:39.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj 2024-03-04T16:35:39.556 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU= 2024-03-04T16:35:39.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CUjigpGMPIF1a0Ep/WeD/aHcvcU=', 'Content-Length': '0'} 2024-03-04T16:35:39.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c19f1e1c6469dbe-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:35:39.559 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.559 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.559 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.559 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.559 DEBUG:boto:Method: GET 2024-03-04T16:35:39.559 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.559 DEBUG:boto:Data: 2024-03-04T16:35:39.559 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.559 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.560 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.560 DEBUG:boto:Params: {} 2024-03-04T16:35:39.560 DEBUG:boto:Token: None 2024-03-04T16:35:39.560 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.560 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000428f9cf53ad608b0-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.562 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:39.562 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:39.562 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.562 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.562 DEBUG:boto:Method: GET 2024-03-04T16:35:39.562 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.562 DEBUG:boto:Data: 2024-03-04T16:35:39.562 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.563 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.563 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.563 DEBUG:boto:Params: {} 2024-03-04T16:35:39.563 DEBUG:boto:Token: None 2024-03-04T16:35:39.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:35:39.563 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8= 2024-03-04T16:35:39.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ITG0xft7oMkn1RXMf1UlVoqUy/8=', 'Content-Length': '0'} 2024-03-04T16:35:39.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082f674d47908d4cd-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.565 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.565 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.565 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.565 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.565 DEBUG:boto:Method: GET 2024-03-04T16:35:39.566 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.566 DEBUG:boto:Data: 2024-03-04T16:35:39.566 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.566 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:35:39.566 DEBUG:boto:Port: 8001 2024-03-04T16:35:39.566 DEBUG:boto:Params: {} 2024-03-04T16:35:39.566 DEBUG:boto:Token: None 2024-03-04T16:35:39.566 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000847b3051d557ff7d-0065e5f85b-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.569 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.569 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.569 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.569 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.569 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:35:39.570 DEBUG:boto:Method: GET 2024-03-04T16:35:39.570 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:35:39.570 DEBUG:boto:Data: 2024-03-04T16:35:39.570 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:35:39.570 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.570 DEBUG:boto:Params: {} 2024-03-04T16:35:39.570 DEBUG:boto:Token: None 2024-03-04T16:35:39.570 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/ 2024-03-04T16:35:39.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4= 2024-03-04T16:35:39.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u8FGwALeLm7cKgIGAZQGR9O2mE4=', 'Content-Length': '0'} 2024-03-04T16:35:39.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8216a8cb3316bc2-0065e5f85b-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.616 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:35:39.616 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:35:39.617 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:39.617 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:39.617 DEBUG:boto:path=/sbnvju-23/?versioning 2024-03-04T16:35:39.617 DEBUG:boto:auth_path=/sbnvju-23/?versioning 2024-03-04T16:35:39.617 DEBUG:boto:Method: PUT 2024-03-04T16:35:39.617 DEBUG:boto:Path: /sbnvju-23/?versioning 2024-03-04T16:35:39.617 DEBUG:boto:Data: Suspended Disabled 2024-03-04T16:35:39.617 DEBUG:boto:Headers: {} 2024-03-04T16:35:39.617 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:39.618 DEBUG:boto:Port: 8000 2024-03-04T16:35:39.618 DEBUG:boto:Params: {} 2024-03-04T16:35:39.618 DEBUG:boto:Token: None 2024-03-04T16:35:39.618 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:35:39 GMT /sbnvju-23/?versioning 2024-03-04T16:35:39.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5kfyScoqYdJ8m+0zJJaXThZgYWo= 2024-03-04T16:35:39.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5kfyScoqYdJ8m+0zJJaXThZgYWo=', 'Content-Length': '230'} 2024-03-04T16:35:39.624 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e288607044c8eadf-0065e5f85b-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:39 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:39.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.625 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:35:39.676 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:39.676 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:39.725 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:35:39.745 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570139.634660_318.1","last_update":"2024-03-04T16:35:39.634660Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:35:39.745 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:39.746 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:35:39.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.746 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:35:39.799 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:39.799 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:39.862+0000 7f2cbc285800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:39.863+0000 7f2cbc285800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:39.872 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.634660_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.634660Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:39Z"} 2024-03-04T16:35:39.873 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.634660_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.634660Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:39Z"} 2024-03-04T16:35:39.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570139.634660_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:35:39.634660Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:35:39.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:39.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:39.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:35:39.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:35:39.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:39.874 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:35:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:39.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:39.978+0000 7feb60533800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:39.979+0000 7feb60533800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:39.988 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.631247_317.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.631247Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:39Z"} 2024-03-04T16:35:39.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.631247_317.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.631247Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:39Z"} 2024-03-04T16:35:39.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570139.631247_317.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:35:39.631247Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:35:39.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:39.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570139.631247_317.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:39.989 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1709570139.634660_318.1 target=1_1709570139.631247_317.1 2024-03-04T16:35:44.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:44.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:35:44.990 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:35:45.045 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:45.107+0000 7f55aff33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:35:45.108+0000 7f55aff33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:35:45.118 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.634660_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.634660Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:45Z"} 2024-03-04T16:35:45.118 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570139.634660_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:35:39.634660Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:35:45Z"} 2024-03-04T16:35:45.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570139.634660_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:35:39.634660Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:35:45.118 DEBUG:rgw_multi.tests:log_status={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:45.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570139.634660_318.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:35:45.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:35:45.119 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:35:45.119 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:35:45.119 DEBUG:boto:Method: HEAD 2024-03-04T16:35:45.119 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:35:45.119 DEBUG:boto:Data: 2024-03-04T16:35:45.119 DEBUG:boto:Headers: {} 2024-03-04T16:35:45.119 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:45.119 DEBUG:boto:Port: 8000 2024-03-04T16:35:45.119 DEBUG:boto:Params: {} 2024-03-04T16:35:45.120 DEBUG:boto:Token: None 2024-03-04T16:35:45.120 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:35:45 GMT /sbnvju-23/ 2024-03-04T16:35:45.120 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Lb7z7w7x0qVJ5ds+KwVtJxCyfNY= 2024-03-04T16:35:45.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:35:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Lb7z7w7x0qVJ5ds+KwVtJxCyfNY=', 'Content-Length': '0'} 2024-03-04T16:35:45.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3d14066ce74bc88-0065e5f861-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:45.123 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:35:45.123 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:35:45.123 DEBUG:boto:Method: PUT 2024-03-04T16:35:45.123 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:35:45.123 DEBUG:boto:Data: 2024-03-04T16:35:45.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:35:45.123 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:35:45.123 DEBUG:boto:Port: 8000 2024-03-04T16:35:45.123 DEBUG:boto:Params: {} 2024-03-04T16:35:45.123 DEBUG:boto:Token: None 2024-03-04T16:35:45.124 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:35:45 GMT /sbnvju-23/obj 2024-03-04T16:35:45.124 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WPETE/JEuXdZnNkZ7kAr47AiwAY= 2024-03-04T16:35:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:35:45 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WPETE/JEuXdZnNkZ7kAr47AiwAY='} 2024-03-04T16:35:45.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d600ba11bb706b34-0065e5f861-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:35:45.136 DEBUG:rgw_multi.tests:created null version id=None 2024-03-04T16:35:45.136 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:35:45.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:45.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:35:45.136 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:35:45.195 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:35:45.196 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:35:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:45.260+0000 7fbf93cc0800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:45.249 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:45.249 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:35:50.251 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:50.261+0000 7fbf93cc0800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:50.251 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:50.251 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:35:55.252 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:35:55.262+0000 7fbf93cc0800 1 waiting for incremental sync to catch up: 2024-03-04T16:35:55.252 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:35:55.252 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:00.253 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:00.264+0000 7fbf93cc0800 1 bucket sync caught up with source: 2024-03-04T16:36:00.254 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:00.254 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:00.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:00.264+0000 7fbf93cc0800 0 bucket checkpoint complete 2024-03-04T16:36:00.276 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:36:00.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:00.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:00.276 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:00.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:00.410+0000 7fd72ca8c800 1 waiting for incremental sync to catch up: 2024-03-04T16:36:00.400 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2024-03-04T16:36:00.400 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:05.411+0000 7fd72ca8c800 1 bucket sync caught up with source: 2024-03-04T16:36:05.401 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:05.401 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2024-03-04T16:36:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:05.411+0000 7fd72ca8c800 0 bucket checkpoint complete 2024-03-04T16:36:05.425 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:36:05.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:05.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:05.425 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:05.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:05.551+0000 7fd8c833b800 1 waiting for incremental sync to catch up: 2024-03-04T16:36:05.540 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:36:05.540 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:10.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:10.552+0000 7fd8c833b800 1 waiting for incremental sync to catch up: 2024-03-04T16:36:10.542 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000004.34.10, , ] 2024-03-04T16:36:10.542 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:15.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:15.554+0000 7fd8c833b800 1 bucket sync caught up with source: 2024-03-04T16:36:15.543 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:15.543 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:15.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:15.554+0000 7fd8c833b800 0 bucket checkpoint complete 2024-03-04T16:36:15.565 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-23 2024-03-04T16:36:15.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:15.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:15.565 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:15.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:15.686+0000 7f9fa7d33800 1 waiting for incremental sync to catch up: 2024-03-04T16:36:15.675 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:15.675 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:20.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:20.688+0000 7f9fa7d33800 1 waiting for incremental sync to catch up: 2024-03-04T16:36:20.677 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000006.39.18, , ] 2024-03-04T16:36:20.677 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:25.689+0000 7f9fa7d33800 1 bucket sync caught up with source: 2024-03-04T16:36:25.678 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.679 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:25.689+0000 7f9fa7d33800 0 bucket checkpoint complete 2024-03-04T16:36:25.703 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-23 2024-03-04T16:36:25.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:25.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:25.703 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:25.763 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:25.763 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:25.843+0000 7f7caca44800 1 bucket sync caught up with source: 2024-03-04T16:36:25.832 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.832 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:25.843+0000 7f7caca44800 0 bucket checkpoint complete 2024-03-04T16:36:25.857 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-23 2024-03-04T16:36:25.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:25.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:25.857 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:25.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:25.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:25.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:25.982+0000 7f3bc8b42800 1 bucket sync caught up with source: 2024-03-04T16:36:25.971 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.972 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000007.41.15, , ] 2024-03-04T16:36:25.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:25.982+0000 7f3bc8b42800 0 bucket checkpoint complete 2024-03-04T16:36:25.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:36:25.990 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:25.990 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:25.991 DEBUG:boto:Method: HEAD 2024-03-04T16:36:25.991 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:25.991 DEBUG:boto:Data: 2024-03-04T16:36:25.991 DEBUG:boto:Headers: {} 2024-03-04T16:36:25.991 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:25.991 DEBUG:boto:Port: 8000 2024-03-04T16:36:25.991 DEBUG:boto:Params: {} 2024-03-04T16:36:25.991 DEBUG:boto:Token: None 2024-03-04T16:36:25.991 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:25 GMT /sbnvju-23/ 2024-03-04T16:36:25.991 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w4bkcg2jAc1UMniOXlg3u/uqrV8= 2024-03-04T16:36:25.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:25 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w4bkcg2jAc1UMniOXlg3u/uqrV8=', 'Content-Length': '0'} 2024-03-04T16:36:25.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff919a0d62293d3-0065e5f88a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:25.994 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:25.994 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:25.995 DEBUG:boto:Method: HEAD 2024-03-04T16:36:25.995 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:25.995 DEBUG:boto:Data: 2024-03-04T16:36:25.995 DEBUG:boto:Headers: {} 2024-03-04T16:36:25.995 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:25.995 DEBUG:boto:Port: 8000 2024-03-04T16:36:25.995 DEBUG:boto:Params: {} 2024-03-04T16:36:25.995 DEBUG:boto:Token: None 2024-03-04T16:36:25.995 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:25 GMT /sbnvju-23/ 2024-03-04T16:36:25.995 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w4bkcg2jAc1UMniOXlg3u/uqrV8= 2024-03-04T16:36:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:25 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w4bkcg2jAc1UMniOXlg3u/uqrV8=', 'Content-Length': '0'} 2024-03-04T16:36:25.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a2f272292aa50c1-0065e5f88a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:25.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:36:25.998 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:25.998 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:25.999 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:25.999 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:25.999 DEBUG:boto:Method: GET 2024-03-04T16:36:25.999 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:25.999 DEBUG:boto:Data: 2024-03-04T16:36:25.999 DEBUG:boto:Headers: {} 2024-03-04T16:36:25.999 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:25.999 DEBUG:boto:Port: 8000 2024-03-04T16:36:25.999 DEBUG:boto:Params: {} 2024-03-04T16:36:25.999 DEBUG:boto:Token: None 2024-03-04T16:36:25.999 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:25 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.000 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/A8XhcfeSGldtTb6IRI+Eq++zEY= 2024-03-04T16:36:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:25 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/A8XhcfeSGldtTb6IRI+Eq++zEY=', 'Content-Length': '0'} 2024-03-04T16:36:26.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb489a3a49f75f2-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.002 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:36:26.003 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.003 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.003 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.003 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.003 DEBUG:boto:Method: GET 2024-03-04T16:36:26.004 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.004 DEBUG:boto:Data: 2024-03-04T16:36:26.004 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.004 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.004 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.004 DEBUG:boto:Params: {} 2024-03-04T16:36:26.004 DEBUG:boto:Token: None 2024-03-04T16:36:26.004 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.004 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a28ef15a255cc6-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.007 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.008 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.008 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.008 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.008 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.009 DEBUG:boto:Method: GET 2024-03-04T16:36:26.009 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.009 DEBUG:boto:Data: 2024-03-04T16:36:26.009 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.010 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.010 DEBUG:boto:Params: {} 2024-03-04T16:36:26.010 DEBUG:boto:Token: None 2024-03-04T16:36:26.010 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.011 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f847716854c4f3-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.013 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.014 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.014 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.014 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.014 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.014 DEBUG:boto:Method: GET 2024-03-04T16:36:26.014 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.014 DEBUG:boto:Data: 2024-03-04T16:36:26.014 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.014 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.014 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.014 DEBUG:boto:Params: {} 2024-03-04T16:36:26.015 DEBUG:boto:Token: None 2024-03-04T16:36:26.015 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.015 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9bf8051304085b-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.017 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.018 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.018 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.018 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.018 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.018 DEBUG:boto:Method: GET 2024-03-04T16:36:26.018 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.018 DEBUG:boto:Data: 2024-03-04T16:36:26.018 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.018 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.018 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.019 DEBUG:boto:Params: {} 2024-03-04T16:36:26.019 DEBUG:boto:Token: None 2024-03-04T16:36:26.019 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.019 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad38ad2e1303582a-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.021 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.021 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.021 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.021 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.021 DEBUG:boto:Method: GET 2024-03-04T16:36:26.022 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.022 DEBUG:boto:Data: 2024-03-04T16:36:26.022 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.022 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.022 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.022 DEBUG:boto:Params: {} 2024-03-04T16:36:26.022 DEBUG:boto:Token: None 2024-03-04T16:36:26.022 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.022 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f93768bd64bb392-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.024 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.025 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.025 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.025 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.025 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.025 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.025 DEBUG:boto:Data: 2024-03-04T16:36:26.025 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.025 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.025 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.025 DEBUG:boto:Params: {} 2024-03-04T16:36:26.026 DEBUG:boto:Token: None 2024-03-04T16:36:26.026 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.026 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000de7aa75113174bb-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.028 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.028 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.028 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.028 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.028 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.028 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.028 DEBUG:boto:Data: 2024-03-04T16:36:26.028 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.028 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.029 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.029 DEBUG:boto:Params: {} 2024-03-04T16:36:26.029 DEBUG:boto:Token: None 2024-03-04T16:36:26.029 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.029 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000197bb4e42b539357-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.031 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.031 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.032 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.032 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.032 DEBUG:boto:Method: GET 2024-03-04T16:36:26.032 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.032 DEBUG:boto:Data: 2024-03-04T16:36:26.032 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.032 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.032 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.032 DEBUG:boto:Params: {} 2024-03-04T16:36:26.032 DEBUG:boto:Token: None 2024-03-04T16:36:26.032 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.033 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc0b14d586e01449-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.034 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.034 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.035 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.035 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.035 DEBUG:boto:Method: GET 2024-03-04T16:36:26.035 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.035 DEBUG:boto:Data: 2024-03-04T16:36:26.035 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.035 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.035 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.036 DEBUG:boto:Params: {} 2024-03-04T16:36:26.036 DEBUG:boto:Token: None 2024-03-04T16:36:26.036 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.036 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb487ff55c72fe78-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.038 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.038 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.038 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.038 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.038 DEBUG:boto:Method: GET 2024-03-04T16:36:26.038 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.038 DEBUG:boto:Data: 2024-03-04T16:36:26.038 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.039 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.039 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.039 DEBUG:boto:Params: {} 2024-03-04T16:36:26.039 DEBUG:boto:Token: None 2024-03-04T16:36:26.039 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.039 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002382dc7cbaa81935-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.041 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.042 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.042 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.042 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.042 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.042 DEBUG:boto:Method: GET 2024-03-04T16:36:26.042 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.042 DEBUG:boto:Data: 2024-03-04T16:36:26.042 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.042 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.043 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.043 DEBUG:boto:Params: {} 2024-03-04T16:36:26.043 DEBUG:boto:Token: None 2024-03-04T16:36:26.043 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.043 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebf17172c0f1b13-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.045 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.046 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.046 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.046 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.046 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.046 DEBUG:boto:Data: 2024-03-04T16:36:26.046 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.047 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.047 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.047 DEBUG:boto:Params: {} 2024-03-04T16:36:26.047 DEBUG:boto:Token: None 2024-03-04T16:36:26.047 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.047 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ad8c01dccf856dd-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.049 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.049 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.049 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.049 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.049 DEBUG:boto:Data: 2024-03-04T16:36:26.049 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.050 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.050 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.050 DEBUG:boto:Params: {} 2024-03-04T16:36:26.050 DEBUG:boto:Token: None 2024-03-04T16:36:26.050 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.050 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faec966f99629fe5-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.053 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.053 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.053 DEBUG:boto:Method: GET 2024-03-04T16:36:26.053 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.053 DEBUG:boto:Data: 2024-03-04T16:36:26.053 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.053 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.053 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.053 DEBUG:boto:Params: {} 2024-03-04T16:36:26.054 DEBUG:boto:Token: None 2024-03-04T16:36:26.054 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.054 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2628e709f6415e5-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.056 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.056 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.056 DEBUG:boto:Method: GET 2024-03-04T16:36:26.056 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.056 DEBUG:boto:Data: 2024-03-04T16:36:26.056 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.056 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.056 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.056 DEBUG:boto:Params: {} 2024-03-04T16:36:26.057 DEBUG:boto:Token: None 2024-03-04T16:36:26.057 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.057 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bcadbb0d70f2c920-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.059 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.059 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.059 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.059 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.059 DEBUG:boto:Method: GET 2024-03-04T16:36:26.059 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.059 DEBUG:boto:Data: 2024-03-04T16:36:26.059 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.059 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.059 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.059 DEBUG:boto:Params: {} 2024-03-04T16:36:26.059 DEBUG:boto:Token: None 2024-03-04T16:36:26.060 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.060 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9554011b2718d17-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.104 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.105 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.105 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.105 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.105 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.105 DEBUG:boto:Method: GET 2024-03-04T16:36:26.105 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.105 DEBUG:boto:Data: 2024-03-04T16:36:26.105 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.105 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.105 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.105 DEBUG:boto:Params: {} 2024-03-04T16:36:26.106 DEBUG:boto:Token: None 2024-03-04T16:36:26.106 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.106 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff0e799ea26f5cb8-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.108 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.109 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.109 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.109 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.109 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.109 DEBUG:boto:Method: GET 2024-03-04T16:36:26.109 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.109 DEBUG:boto:Data: 2024-03-04T16:36:26.109 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.109 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.110 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.110 DEBUG:boto:Params: {} 2024-03-04T16:36:26.110 DEBUG:boto:Token: None 2024-03-04T16:36:26.110 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.110 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085ccad3b69d1da7c-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.112 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.112 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.112 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.112 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.112 DEBUG:boto:Method: GET 2024-03-04T16:36:26.112 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.113 DEBUG:boto:Data: 2024-03-04T16:36:26.113 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.113 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.113 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.113 DEBUG:boto:Params: {} 2024-03-04T16:36:26.113 DEBUG:boto:Token: None 2024-03-04T16:36:26.113 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.113 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000605f0330fa1a689c-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.115 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.115 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.115 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.116 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.116 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.116 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.116 DEBUG:boto:Data: 2024-03-04T16:36:26.116 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.116 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.116 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.116 DEBUG:boto:Params: {} 2024-03-04T16:36:26.116 DEBUG:boto:Token: None 2024-03-04T16:36:26.116 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.117 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004526f423ec324bef-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.119 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.119 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.119 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.119 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.119 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.120 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.120 DEBUG:boto:Data: 2024-03-04T16:36:26.120 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.120 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.120 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.120 DEBUG:boto:Params: {} 2024-03-04T16:36:26.120 DEBUG:boto:Token: None 2024-03-04T16:36:26.120 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.121 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000decf19f9622ba3a7-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.123 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.123 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.123 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.123 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.123 DEBUG:boto:Method: GET 2024-03-04T16:36:26.123 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.123 DEBUG:boto:Data: 2024-03-04T16:36:26.123 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.123 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.124 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.124 DEBUG:boto:Params: {} 2024-03-04T16:36:26.124 DEBUG:boto:Token: None 2024-03-04T16:36:26.124 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.124 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed2b54b5bae3225f-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.126 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.126 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.126 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.126 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.126 DEBUG:boto:Method: GET 2024-03-04T16:36:26.126 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.127 DEBUG:boto:Data: 2024-03-04T16:36:26.127 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.127 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.127 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.127 DEBUG:boto:Params: {} 2024-03-04T16:36:26.127 DEBUG:boto:Token: None 2024-03-04T16:36:26.127 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.127 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9633406ba4f1850-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.129 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.130 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.130 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.130 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.130 DEBUG:boto:Method: GET 2024-03-04T16:36:26.130 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.130 DEBUG:boto:Data: 2024-03-04T16:36:26.130 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.130 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.130 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.130 DEBUG:boto:Params: {} 2024-03-04T16:36:26.130 DEBUG:boto:Token: None 2024-03-04T16:36:26.131 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.131 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a28a46d5be5f9d2-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.133 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.134 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.134 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.134 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.134 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.134 DEBUG:boto:Method: GET 2024-03-04T16:36:26.134 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.134 DEBUG:boto:Data: 2024-03-04T16:36:26.134 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.134 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.134 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.135 DEBUG:boto:Params: {} 2024-03-04T16:36:26.135 DEBUG:boto:Token: None 2024-03-04T16:36:26.135 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.135 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a5119a91ba8080-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.137 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.138 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.138 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.138 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.138 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.138 DEBUG:boto:Data: 2024-03-04T16:36:26.138 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.139 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.139 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.139 DEBUG:boto:Params: {} 2024-03-04T16:36:26.139 DEBUG:boto:Token: None 2024-03-04T16:36:26.139 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.139 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e29bd01ebc8d7ee4-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.143 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.143 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.143 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.143 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.143 DEBUG:boto:Data: 2024-03-04T16:36:26.143 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.143 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.143 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.143 DEBUG:boto:Params: {} 2024-03-04T16:36:26.144 DEBUG:boto:Token: None 2024-03-04T16:36:26.144 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.144 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc2d985afa280da3-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.146 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.147 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.147 DEBUG:boto:Method: GET 2024-03-04T16:36:26.147 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.147 DEBUG:boto:Data: 2024-03-04T16:36:26.147 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.147 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.147 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.147 DEBUG:boto:Params: {} 2024-03-04T16:36:26.147 DEBUG:boto:Token: None 2024-03-04T16:36:26.148 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.148 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e34a96a4deb2afa-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.150 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.150 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.150 DEBUG:boto:Method: GET 2024-03-04T16:36:26.150 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.150 DEBUG:boto:Data: 2024-03-04T16:36:26.150 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.150 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.150 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.150 DEBUG:boto:Params: {} 2024-03-04T16:36:26.150 DEBUG:boto:Token: None 2024-03-04T16:36:26.151 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.151 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080e2132ff12d29f8-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.152 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.153 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.153 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.153 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.153 DEBUG:boto:Method: GET 2024-03-04T16:36:26.153 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.153 DEBUG:boto:Data: 2024-03-04T16:36:26.153 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.153 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.153 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.153 DEBUG:boto:Params: {} 2024-03-04T16:36:26.153 DEBUG:boto:Token: None 2024-03-04T16:36:26.154 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.154 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000532df5888ff2010-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.204 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.205 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.205 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.205 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.205 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.205 DEBUG:boto:Method: GET 2024-03-04T16:36:26.205 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.205 DEBUG:boto:Data: 2024-03-04T16:36:26.206 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.206 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.206 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.206 DEBUG:boto:Params: {} 2024-03-04T16:36:26.206 DEBUG:boto:Token: None 2024-03-04T16:36:26.206 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.206 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b5f989034e0741-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.209 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone2, test-zone1} 2024-03-04T16:36:26.209 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:36:26.210 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.210 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.210 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.210 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:26.210 DEBUG:boto:Data: 2024-03-04T16:36:26.210 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.210 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.210 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.210 DEBUG:boto:Params: {} 2024-03-04T16:36:26.211 DEBUG:boto:Token: None 2024-03-04T16:36:26.211 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.211 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk= 2024-03-04T16:36:26.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk=', 'Content-Length': '0'} 2024-03-04T16:36:26.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4693a465f3f6010-0065e5f88a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.213 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.213 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.213 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.214 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:26.214 DEBUG:boto:Data: 2024-03-04T16:36:26.214 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.214 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.214 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.214 DEBUG:boto:Params: {} 2024-03-04T16:36:26.214 DEBUG:boto:Token: None 2024-03-04T16:36:26.214 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.214 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk= 2024-03-04T16:36:26.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk=', 'Content-Length': '0'} 2024-03-04T16:36:26.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062e9aa00298ad1b3-0065e5f88a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:36:26.217 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.217 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.217 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.217 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.217 DEBUG:boto:Method: GET 2024-03-04T16:36:26.217 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.218 DEBUG:boto:Data: 2024-03-04T16:36:26.218 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.218 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.218 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.218 DEBUG:boto:Params: {} 2024-03-04T16:36:26.218 DEBUG:boto:Token: None 2024-03-04T16:36:26.218 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.218 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018159096ff396b00-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.268 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:36:26.269 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.269 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.269 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.269 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.269 DEBUG:boto:Method: GET 2024-03-04T16:36:26.269 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.270 DEBUG:boto:Data: 2024-03-04T16:36:26.270 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.270 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.270 DEBUG:boto:Params: {} 2024-03-04T16:36:26.270 DEBUG:boto:Token: None 2024-03-04T16:36:26.270 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.270 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e259181d07cfaa-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.273 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.274 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.274 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.274 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.274 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.274 DEBUG:boto:Method: GET 2024-03-04T16:36:26.274 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.274 DEBUG:boto:Data: 2024-03-04T16:36:26.274 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.274 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.275 DEBUG:boto:Params: {} 2024-03-04T16:36:26.275 DEBUG:boto:Token: None 2024-03-04T16:36:26.275 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc350b0ccc0a8d0-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.278 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.278 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.278 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.279 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.279 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.279 DEBUG:boto:Method: GET 2024-03-04T16:36:26.279 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.279 DEBUG:boto:Data: 2024-03-04T16:36:26.279 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.279 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.279 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.279 DEBUG:boto:Params: {} 2024-03-04T16:36:26.279 DEBUG:boto:Token: None 2024-03-04T16:36:26.280 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f5a166089886cb-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.282 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.282 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.283 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.283 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.283 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.283 DEBUG:boto:Method: GET 2024-03-04T16:36:26.283 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.283 DEBUG:boto:Data: 2024-03-04T16:36:26.283 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.283 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.283 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.283 DEBUG:boto:Params: {} 2024-03-04T16:36:26.284 DEBUG:boto:Token: None 2024-03-04T16:36:26.284 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.284 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034b3191ee1a0d950-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.286 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.286 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.286 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.286 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.286 DEBUG:boto:Method: GET 2024-03-04T16:36:26.286 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.286 DEBUG:boto:Data: 2024-03-04T16:36:26.287 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.287 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.287 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.287 DEBUG:boto:Params: {} 2024-03-04T16:36:26.287 DEBUG:boto:Token: None 2024-03-04T16:36:26.287 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.287 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070b5682668bd8a7b-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.289 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.289 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.289 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.289 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.289 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.290 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.290 DEBUG:boto:Data: 2024-03-04T16:36:26.290 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.290 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.290 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.290 DEBUG:boto:Params: {} 2024-03-04T16:36:26.290 DEBUG:boto:Token: None 2024-03-04T16:36:26.290 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.290 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000981ccd66b5717c58-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.292 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.293 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.293 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.293 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.293 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.293 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.293 DEBUG:boto:Data: 2024-03-04T16:36:26.293 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.293 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.293 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.293 DEBUG:boto:Params: {} 2024-03-04T16:36:26.293 DEBUG:boto:Token: None 2024-03-04T16:36:26.294 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.294 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a07fbb5f41d07847-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.296 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.296 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.296 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.297 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.297 DEBUG:boto:Method: GET 2024-03-04T16:36:26.297 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.297 DEBUG:boto:Data: 2024-03-04T16:36:26.297 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.297 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.297 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.297 DEBUG:boto:Params: {} 2024-03-04T16:36:26.297 DEBUG:boto:Token: None 2024-03-04T16:36:26.297 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.298 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058fe304a4c3176ea-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.300 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.300 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.300 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.300 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.300 DEBUG:boto:Method: GET 2024-03-04T16:36:26.300 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.300 DEBUG:boto:Data: 2024-03-04T16:36:26.300 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.301 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.301 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.301 DEBUG:boto:Params: {} 2024-03-04T16:36:26.301 DEBUG:boto:Token: None 2024-03-04T16:36:26.301 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.301 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a90ba05e506f2fc1-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.303 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.303 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.303 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.303 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.303 DEBUG:boto:Method: GET 2024-03-04T16:36:26.303 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.303 DEBUG:boto:Data: 2024-03-04T16:36:26.303 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.304 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.304 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.304 DEBUG:boto:Params: {} 2024-03-04T16:36:26.304 DEBUG:boto:Token: None 2024-03-04T16:36:26.304 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.304 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e8c62c2ab5241b-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.348 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.349 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.349 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.349 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.349 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.349 DEBUG:boto:Method: GET 2024-03-04T16:36:26.349 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.349 DEBUG:boto:Data: 2024-03-04T16:36:26.349 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.349 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.349 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.349 DEBUG:boto:Params: {} 2024-03-04T16:36:26.350 DEBUG:boto:Token: None 2024-03-04T16:36:26.350 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.350 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504344399f5c7e0b-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.352 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.353 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.353 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.353 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.353 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.353 DEBUG:boto:Data: 2024-03-04T16:36:26.353 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.353 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.353 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.353 DEBUG:boto:Params: {} 2024-03-04T16:36:26.353 DEBUG:boto:Token: None 2024-03-04T16:36:26.354 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.354 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009afd2b8e5a7aed36-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.355 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.355 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.355 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.356 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.356 DEBUG:boto:Data: 2024-03-04T16:36:26.356 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.356 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.356 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.356 DEBUG:boto:Params: {} 2024-03-04T16:36:26.356 DEBUG:boto:Token: None 2024-03-04T16:36:26.356 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.356 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b88de699f6084102-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.358 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.358 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.358 DEBUG:boto:Method: GET 2024-03-04T16:36:26.358 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.359 DEBUG:boto:Data: 2024-03-04T16:36:26.359 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.359 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.359 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.359 DEBUG:boto:Params: {} 2024-03-04T16:36:26.359 DEBUG:boto:Token: None 2024-03-04T16:36:26.359 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.359 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa2842a769c2cda6-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.361 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.361 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.361 DEBUG:boto:Method: GET 2024-03-04T16:36:26.361 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.361 DEBUG:boto:Data: 2024-03-04T16:36:26.361 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.361 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.361 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.362 DEBUG:boto:Params: {} 2024-03-04T16:36:26.362 DEBUG:boto:Token: None 2024-03-04T16:36:26.362 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.362 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a54896c26c23d273-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.364 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.364 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.364 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.364 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.364 DEBUG:boto:Method: GET 2024-03-04T16:36:26.364 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.364 DEBUG:boto:Data: 2024-03-04T16:36:26.364 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.364 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.365 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.365 DEBUG:boto:Params: {} 2024-03-04T16:36:26.365 DEBUG:boto:Token: None 2024-03-04T16:36:26.365 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.365 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f16844ec25e6a6-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.367 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.367 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.368 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.368 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.368 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.368 DEBUG:boto:Method: GET 2024-03-04T16:36:26.368 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.368 DEBUG:boto:Data: 2024-03-04T16:36:26.368 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.368 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.368 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.368 DEBUG:boto:Params: {} 2024-03-04T16:36:26.369 DEBUG:boto:Token: None 2024-03-04T16:36:26.369 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.369 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000266400dfc55b49c7-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.412 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.413 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.413 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.413 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.413 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.413 DEBUG:boto:Method: GET 2024-03-04T16:36:26.413 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.414 DEBUG:boto:Data: 2024-03-04T16:36:26.414 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.414 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.414 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.414 DEBUG:boto:Params: {} 2024-03-04T16:36:26.414 DEBUG:boto:Token: None 2024-03-04T16:36:26.414 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.415 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6608f0f812100d4-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.417 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.417 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.417 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.417 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.417 DEBUG:boto:Method: GET 2024-03-04T16:36:26.417 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.417 DEBUG:boto:Data: 2024-03-04T16:36:26.417 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.418 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.418 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.418 DEBUG:boto:Params: {} 2024-03-04T16:36:26.418 DEBUG:boto:Token: None 2024-03-04T16:36:26.418 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.418 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096857ab250613509-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.420 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.420 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.420 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.420 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.420 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.421 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.421 DEBUG:boto:Data: 2024-03-04T16:36:26.421 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.421 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.421 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.421 DEBUG:boto:Params: {} 2024-03-04T16:36:26.421 DEBUG:boto:Token: None 2024-03-04T16:36:26.421 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.421 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007019fe9934cd9e74-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.423 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.423 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.424 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.424 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.424 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.424 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.424 DEBUG:boto:Data: 2024-03-04T16:36:26.424 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.424 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.424 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.424 DEBUG:boto:Params: {} 2024-03-04T16:36:26.425 DEBUG:boto:Token: None 2024-03-04T16:36:26.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c81ebc35934e88e-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.427 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.427 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.427 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.427 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.428 DEBUG:boto:Method: GET 2024-03-04T16:36:26.428 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.428 DEBUG:boto:Data: 2024-03-04T16:36:26.428 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.428 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.428 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.428 DEBUG:boto:Params: {} 2024-03-04T16:36:26.429 DEBUG:boto:Token: None 2024-03-04T16:36:26.429 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095a401c1ae512d1d-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.431 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.431 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.431 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.431 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.431 DEBUG:boto:Method: GET 2024-03-04T16:36:26.431 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.431 DEBUG:boto:Data: 2024-03-04T16:36:26.432 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.432 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.432 DEBUG:boto:Params: {} 2024-03-04T16:36:26.432 DEBUG:boto:Token: None 2024-03-04T16:36:26.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a01eb4b5ff6f8d5-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.434 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.434 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.434 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.434 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.434 DEBUG:boto:Method: GET 2024-03-04T16:36:26.434 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.434 DEBUG:boto:Data: 2024-03-04T16:36:26.435 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.435 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.435 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.435 DEBUG:boto:Params: {} 2024-03-04T16:36:26.435 DEBUG:boto:Token: None 2024-03-04T16:36:26.435 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.435 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a537992835747438-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.480 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.481 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.481 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.481 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.481 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.481 DEBUG:boto:Method: GET 2024-03-04T16:36:26.481 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.481 DEBUG:boto:Data: 2024-03-04T16:36:26.481 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.481 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.481 DEBUG:boto:Params: {} 2024-03-04T16:36:26.482 DEBUG:boto:Token: None 2024-03-04T16:36:26.482 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a5cf44a9015dce-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.484 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.485 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.485 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.485 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.485 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.485 DEBUG:boto:Data: 2024-03-04T16:36:26.485 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.485 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.485 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.485 DEBUG:boto:Params: {} 2024-03-04T16:36:26.485 DEBUG:boto:Token: None 2024-03-04T16:36:26.486 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009418779fd44bd7b3-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.487 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.487 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.487 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.488 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.488 DEBUG:boto:Data: 2024-03-04T16:36:26.488 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.488 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.488 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.488 DEBUG:boto:Params: {} 2024-03-04T16:36:26.488 DEBUG:boto:Token: None 2024-03-04T16:36:26.488 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ba3aed21fc752e2-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.490 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.490 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.490 DEBUG:boto:Method: GET 2024-03-04T16:36:26.491 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.491 DEBUG:boto:Data: 2024-03-04T16:36:26.491 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.491 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.491 DEBUG:boto:Params: {} 2024-03-04T16:36:26.491 DEBUG:boto:Token: None 2024-03-04T16:36:26.491 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050261062a5647cc7-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.493 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.493 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.493 DEBUG:boto:Method: GET 2024-03-04T16:36:26.493 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.493 DEBUG:boto:Data: 2024-03-04T16:36:26.493 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.494 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.494 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.494 DEBUG:boto:Params: {} 2024-03-04T16:36:26.494 DEBUG:boto:Token: None 2024-03-04T16:36:26.494 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061624c18f35c34b6-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.496 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.496 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.496 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.496 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.496 DEBUG:boto:Method: GET 2024-03-04T16:36:26.497 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.497 DEBUG:boto:Data: 2024-03-04T16:36:26.497 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.497 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.497 DEBUG:boto:Params: {} 2024-03-04T16:36:26.497 DEBUG:boto:Token: None 2024-03-04T16:36:26.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9a34ab43870c148-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.500 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.500 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.500 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.500 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.500 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.500 DEBUG:boto:Method: GET 2024-03-04T16:36:26.500 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.501 DEBUG:boto:Data: 2024-03-04T16:36:26.501 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.501 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.501 DEBUG:boto:Params: {} 2024-03-04T16:36:26.501 DEBUG:boto:Token: None 2024-03-04T16:36:26.501 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc672b5a810aacd-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.505 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.505 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone1} 2024-03-04T16:36:26.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:36:26.505 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.505 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.505 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.505 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:26.506 DEBUG:boto:Data: 2024-03-04T16:36:26.506 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.506 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.506 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.506 DEBUG:boto:Params: {} 2024-03-04T16:36:26.506 DEBUG:boto:Token: None 2024-03-04T16:36:26.506 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk= 2024-03-04T16:36:26.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk=', 'Content-Length': '0'} 2024-03-04T16:36:26.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa4622963c25dd9-0065e5f88a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.509 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.509 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.509 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.509 DEBUG:boto:Path: /sbnvju-23/ 2024-03-04T16:36:26.509 DEBUG:boto:Data: 2024-03-04T16:36:26.510 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.510 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.510 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.510 DEBUG:boto:Params: {} 2024-03-04T16:36:26.510 DEBUG:boto:Token: None 2024-03-04T16:36:26.510 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk= 2024-03-04T16:36:26.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MLC+AUYK3QqbMda4gU1d1T+wrhk=', 'Content-Length': '0'} 2024-03-04T16:36:26.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001d27473386cc5c2-0065e5f88a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.513 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:36:26.513 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.513 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.513 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.513 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.513 DEBUG:boto:Method: GET 2024-03-04T16:36:26.514 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.514 DEBUG:boto:Data: 2024-03-04T16:36:26.514 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.514 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.514 DEBUG:boto:Params: {} 2024-03-04T16:36:26.514 DEBUG:boto:Token: None 2024-03-04T16:36:26.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.514 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002edbc96a839a6ba5-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.517 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.517 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:36:26.517 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.517 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.518 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.518 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.518 DEBUG:boto:Method: GET 2024-03-04T16:36:26.518 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.518 DEBUG:boto:Data: 2024-03-04T16:36:26.518 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.518 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.518 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.518 DEBUG:boto:Params: {} 2024-03-04T16:36:26.518 DEBUG:boto:Token: None 2024-03-04T16:36:26.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009caacb7f2f837d99-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.521 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:36:26.522 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.522 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.522 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.522 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.522 DEBUG:boto:Method: GET 2024-03-04T16:36:26.522 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.522 DEBUG:boto:Data: 2024-03-04T16:36:26.522 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.522 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.522 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.522 DEBUG:boto:Params: {} 2024-03-04T16:36:26.523 DEBUG:boto:Token: None 2024-03-04T16:36:26.523 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.523 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3e72f16ae0210f-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.572 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.572 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.572 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.572 DEBUG:boto:path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.573 DEBUG:boto:auth_path=/sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.573 DEBUG:boto:Method: GET 2024-03-04T16:36:26.573 DEBUG:boto:Path: /sbnvju-23/?versions&max-keys=999 2024-03-04T16:36:26.573 DEBUG:boto:Data: 2024-03-04T16:36:26.573 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.573 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.573 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.573 DEBUG:boto:Params: {} 2024-03-04T16:36:26.573 DEBUG:boto:Token: None 2024-03-04T16:36:26.573 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/?versions 2024-03-04T16:36:26.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA= 2024-03-04T16:36:26.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wa0yihGMeFQp7Yr2o0X3+pWwpdA=', 'Content-Length': '0'} 2024-03-04T16:36:26.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001534c0b39fdbe5c9-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.576 DEBUG:boto:b'sbnvju-23999falseobjnulltrue2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPy8vSx1bR6JHgSIzxtfKKS82mw0pim1false2024-03-04T16:34:38.077Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.576 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.576 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.577 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.577 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.577 DEBUG:boto:Method: GET 2024-03-04T16:36:26.577 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.577 DEBUG:boto:Data: 2024-03-04T16:36:26.577 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.577 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.577 DEBUG:boto:Params: {} 2024-03-04T16:36:26.577 DEBUG:boto:Token: None 2024-03-04T16:36:26.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023cb6e717252a952-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.580 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.580 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.580 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.580 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.580 DEBUG:boto:Method: GET 2024-03-04T16:36:26.580 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.580 DEBUG:boto:Data: 2024-03-04T16:36:26.580 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.580 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.580 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.581 DEBUG:boto:Params: {} 2024-03-04T16:36:26.581 DEBUG:boto:Token: None 2024-03-04T16:36:26.581 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da3b03da480b5cff-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.583 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.583 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.583 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.583 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.584 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.584 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.584 DEBUG:boto:Data: 2024-03-04T16:36:26.584 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.584 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.584 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.584 DEBUG:boto:Params: {} 2024-03-04T16:36:26.584 DEBUG:boto:Token: None 2024-03-04T16:36:26.584 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004366881a734a32e4-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.586 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.586 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.586 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.586 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.587 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.587 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.587 DEBUG:boto:Data: 2024-03-04T16:36:26.587 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.587 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.587 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.587 DEBUG:boto:Params: {} 2024-03-04T16:36:26.587 DEBUG:boto:Token: None 2024-03-04T16:36:26.587 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.587 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k= 2024-03-04T16:36:26.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BvjmnKkeJJyUmRX2ng13wV2MC8k=', 'Content-Length': '0'} 2024-03-04T16:36:26.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005fc6e1ee751b81a-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.590 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.590 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.590 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.590 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.590 DEBUG:boto:Method: GET 2024-03-04T16:36:26.590 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.590 DEBUG:boto:Data: 2024-03-04T16:36:26.590 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.590 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.590 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.591 DEBUG:boto:Params: {} 2024-03-04T16:36:26.591 DEBUG:boto:Token: None 2024-03-04T16:36:26.591 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.591 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8ff5435a1e9cb27-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.593 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.593 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.593 DEBUG:boto:path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.593 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.593 DEBUG:boto:Method: GET 2024-03-04T16:36:26.593 DEBUG:boto:Path: /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.593 DEBUG:boto:Data: 2024-03-04T16:36:26.593 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.593 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.593 DEBUG:boto:Params: {} 2024-03-04T16:36:26.594 DEBUG:boto:Token: None 2024-03-04T16:36:26.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=null 2024-03-04T16:36:26.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU= 2024-03-04T16:36:26.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ZWU0JBR4F7uTvUGD8kyDE3DcaU=', 'Content-Length': '0'} 2024-03-04T16:36:26.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6ef5a8b3c890c31-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.596 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.596 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.596 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.596 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.596 DEBUG:boto:Method: GET 2024-03-04T16:36:26.596 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.597 DEBUG:boto:Data: 2024-03-04T16:36:26.597 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.597 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.597 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.597 DEBUG:boto:Params: {} 2024-03-04T16:36:26.597 DEBUG:boto:Token: None 2024-03-04T16:36:26.597 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.597 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac31671d0e0e6332-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.600 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.600 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.600 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.600 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.600 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.600 DEBUG:boto:Method: GET 2024-03-04T16:36:26.600 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.601 DEBUG:boto:Data: 2024-03-04T16:36:26.601 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.601 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.601 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.601 DEBUG:boto:Params: {} 2024-03-04T16:36:26.601 DEBUG:boto:Token: None 2024-03-04T16:36:26.601 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739fa73a9ebb0540-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.604 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.604 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.604 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.604 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.604 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.604 DEBUG:boto:Data: 2024-03-04T16:36:26.604 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.605 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.605 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.605 DEBUG:boto:Params: {} 2024-03-04T16:36:26.605 DEBUG:boto:Token: None 2024-03-04T16:36:26.605 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f235103caaa97190-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.607 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.608 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.608 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.608 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.608 DEBUG:boto:Data: 2024-03-04T16:36:26.608 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.608 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.608 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.608 DEBUG:boto:Params: {} 2024-03-04T16:36:26.608 DEBUG:boto:Token: None 2024-03-04T16:36:26.608 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063929d22cecbc1d9-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.610 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.611 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.611 DEBUG:boto:Method: GET 2024-03-04T16:36:26.611 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.611 DEBUG:boto:Data: 2024-03-04T16:36:26.611 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.611 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.611 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.611 DEBUG:boto:Params: {} 2024-03-04T16:36:26.611 DEBUG:boto:Token: None 2024-03-04T16:36:26.611 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.611 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b636149cd8311479-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.613 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.614 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.614 DEBUG:boto:Method: GET 2024-03-04T16:36:26.614 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.614 DEBUG:boto:Data: 2024-03-04T16:36:26.614 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.614 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.614 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.614 DEBUG:boto:Params: {} 2024-03-04T16:36:26.614 DEBUG:boto:Token: None 2024-03-04T16:36:26.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.615 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ceec724a4ed88dc6-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.617 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.617 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.617 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.617 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.617 DEBUG:boto:Method: GET 2024-03-04T16:36:26.617 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.617 DEBUG:boto:Data: 2024-03-04T16:36:26.617 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.617 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.617 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.617 DEBUG:boto:Params: {} 2024-03-04T16:36:26.618 DEBUG:boto:Token: None 2024-03-04T16:36:26.618 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1f2fda4280bf74e-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.620 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.621 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.621 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.621 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.621 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.621 DEBUG:boto:Method: GET 2024-03-04T16:36:26.621 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.621 DEBUG:boto:Data: 2024-03-04T16:36:26.621 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.622 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.622 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.622 DEBUG:boto:Params: {} 2024-03-04T16:36:26.622 DEBUG:boto:Token: None 2024-03-04T16:36:26.622 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.622 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be7d0a690c7d17d-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.625 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.625 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.625 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.625 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.625 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.625 DEBUG:boto:Method: GET 2024-03-04T16:36:26.626 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.626 DEBUG:boto:Data: 2024-03-04T16:36:26.626 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.626 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.626 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.626 DEBUG:boto:Params: {} 2024-03-04T16:36:26.626 DEBUG:boto:Token: None 2024-03-04T16:36:26.626 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.626 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b174723fe22289b6-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.628 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.628 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.628 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.628 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.628 DEBUG:boto:Method: GET 2024-03-04T16:36:26.629 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.629 DEBUG:boto:Data: 2024-03-04T16:36:26.629 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.629 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.629 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.629 DEBUG:boto:Params: {} 2024-03-04T16:36:26.629 DEBUG:boto:Token: None 2024-03-04T16:36:26.629 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.629 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f451726d4d1ce8e5-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.632 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.632 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.632 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.632 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.632 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.632 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.632 DEBUG:boto:Data: 2024-03-04T16:36:26.632 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.632 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.632 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.633 DEBUG:boto:Params: {} 2024-03-04T16:36:26.633 DEBUG:boto:Token: None 2024-03-04T16:36:26.633 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.633 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fbcef3f9b0cb3c5-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.635 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.635 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.635 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.635 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.635 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.635 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.635 DEBUG:boto:Data: 2024-03-04T16:36:26.636 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.636 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.636 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.636 DEBUG:boto:Params: {} 2024-03-04T16:36:26.636 DEBUG:boto:Token: None 2024-03-04T16:36:26.636 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.636 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0= 2024-03-04T16:36:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2GZYekGubJW+nbBT+or2t4qAIc0=', 'Content-Length': '0'} 2024-03-04T16:36:26.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d7b66b2eb3b11b2-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.638 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.639 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.639 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.639 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.639 DEBUG:boto:Method: GET 2024-03-04T16:36:26.639 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.639 DEBUG:boto:Data: 2024-03-04T16:36:26.639 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.639 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.639 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.639 DEBUG:boto:Params: {} 2024-03-04T16:36:26.639 DEBUG:boto:Token: None 2024-03-04T16:36:26.639 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.640 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000693658f3b93c050c-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.641 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.642 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.642 DEBUG:boto:path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.642 DEBUG:boto:auth_path=/sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.642 DEBUG:boto:Method: GET 2024-03-04T16:36:26.642 DEBUG:boto:Path: /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.642 DEBUG:boto:Data: 2024-03-04T16:36:26.642 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.642 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.642 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.642 DEBUG:boto:Params: {} 2024-03-04T16:36:26.642 DEBUG:boto:Token: None 2024-03-04T16:36:26.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj?versionId=Py8vSx1bR6JHgSIzxtfKKS82mw0pim1 2024-03-04T16:36:26.643 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA= 2024-03-04T16:36:26.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0/93t0fvzemE9ffBwEtctvgaqKA=', 'Content-Length': '0'} 2024-03-04T16:36:26.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:34:38 GMT'), ('x-amz-version-id', 'Py8vSx1bR6JHgSIzxtfKKS82mw0pim1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4a4a94a9a3d693a-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.645 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.645 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.645 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.645 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.645 DEBUG:boto:Method: GET 2024-03-04T16:36:26.646 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.646 DEBUG:boto:Data: 2024-03-04T16:36:26.646 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.646 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.646 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.646 DEBUG:boto:Params: {} 2024-03-04T16:36:26.646 DEBUG:boto:Token: None 2024-03-04T16:36:26.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b611a7e02a053057-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.692 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.692 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.693 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.693 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.693 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.693 DEBUG:boto:Method: GET 2024-03-04T16:36:26.693 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.693 DEBUG:boto:Data: 2024-03-04T16:36:26.693 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.693 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.693 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.693 DEBUG:boto:Params: {} 2024-03-04T16:36:26.693 DEBUG:boto:Token: None 2024-03-04T16:36:26.694 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014423d39407a758d-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.696 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.697 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.697 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.697 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.697 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.697 DEBUG:boto:Data: 2024-03-04T16:36:26.697 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.697 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.697 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.698 DEBUG:boto:Params: {} 2024-03-04T16:36:26.698 DEBUG:boto:Token: None 2024-03-04T16:36:26.698 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.698 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f33498c937671259-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.700 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.700 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.700 DEBUG:boto:Method: HEAD 2024-03-04T16:36:26.700 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.700 DEBUG:boto:Data: 2024-03-04T16:36:26.701 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.701 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.701 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.701 DEBUG:boto:Params: {} 2024-03-04T16:36:26.701 DEBUG:boto:Token: None 2024-03-04T16:36:26.701 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.701 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M= 2024-03-04T16:36:26.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CSBGLNnCmxp6g7GlokJI/19W0+M=', 'Content-Length': '0'} 2024-03-04T16:36:26.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da02c494ef68822e-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:36:26.703 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.703 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.703 DEBUG:boto:Method: GET 2024-03-04T16:36:26.704 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.704 DEBUG:boto:Data: 2024-03-04T16:36:26.704 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.704 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.704 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.704 DEBUG:boto:Params: {} 2024-03-04T16:36:26.704 DEBUG:boto:Token: None 2024-03-04T16:36:26.704 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.704 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7413bfa3a90039b-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.706 DEBUG:boto:path=/sbnvju-23/obj 2024-03-04T16:36:26.706 DEBUG:boto:auth_path=/sbnvju-23/obj 2024-03-04T16:36:26.706 DEBUG:boto:Method: GET 2024-03-04T16:36:26.707 DEBUG:boto:Path: /sbnvju-23/obj 2024-03-04T16:36:26.707 DEBUG:boto:Data: 2024-03-04T16:36:26.707 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.707 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.707 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.707 DEBUG:boto:Params: {} 2024-03-04T16:36:26.707 DEBUG:boto:Token: None 2024-03-04T16:36:26.707 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/obj 2024-03-04T16:36:26.707 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c= 2024-03-04T16:36:26.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oP2iVHBQ4U9v5FIRApeCfb4xo0c=', 'Content-Length': '0'} 2024-03-04T16:36:26.709 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:35:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-23:fc294220-5038-4f25-aeee-cac2c5838729.4341.32'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8833c553dd29fca-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.709 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.710 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.710 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.710 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.710 DEBUG:boto:Method: GET 2024-03-04T16:36:26.710 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.710 DEBUG:boto:Data: 2024-03-04T16:36:26.710 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.710 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.710 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.710 DEBUG:boto:Params: {} 2024-03-04T16:36:26.711 DEBUG:boto:Token: None 2024-03-04T16:36:26.711 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.711 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4ee1a5b848fedf-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.714 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.714 DEBUG:boto:path=/sbnvju-23/ 2024-03-04T16:36:26.714 DEBUG:boto:auth_path=/sbnvju-23/ 2024-03-04T16:36:26.714 DEBUG:boto:path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.714 DEBUG:boto:auth_path=/sbnvju-23/?prefix=obj 2024-03-04T16:36:26.714 DEBUG:boto:Method: GET 2024-03-04T16:36:26.714 DEBUG:boto:Path: /sbnvju-23/?prefix=obj 2024-03-04T16:36:26.714 DEBUG:boto:Data: 2024-03-04T16:36:26.715 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.715 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.715 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.715 DEBUG:boto:Params: {} 2024-03-04T16:36:26.715 DEBUG:boto:Token: None 2024-03-04T16:36:26.715 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-23/ 2024-03-04T16:36:26.715 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00= 2024-03-04T16:36:26.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UaUd8Ir+lfhH4uSnJaOIzdT0V00=', 'Content-Length': '0'} 2024-03-04T16:36:26.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f64931c8a990fc-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.718 DEBUG:boto:b'sbnvju-23obj1000falseobj2024-03-04T16:35:45.137Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:36:26.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-23 zones={test-zone3, test-zone2} 2024-03-04T16:36:26.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-03-04T16:36:26.720 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-24 2024-03-04T16:36:26.720 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:36:26.720 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:36:26.720 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.721 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:36:26.721 DEBUG:boto:Data: 2024-03-04T16:36:26.721 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.721 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.721 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.721 DEBUG:boto:Params: {} 2024-03-04T16:36:26.721 DEBUG:boto:Token: None 2024-03-04T16:36:26.721 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-24/ 2024-03-04T16:36:26.721 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZjdzAbocoF2V4qI4S3Aeflqa7TQ= 2024-03-04T16:36:26.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZjdzAbocoF2V4qI4S3Aeflqa7TQ=', 'Content-Length': '0'} 2024-03-04T16:36:26.735 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090001b332df554fd-0065e5f88a-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.735 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-25 2024-03-04T16:36:26.736 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:36:26.736 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:36:26.736 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.736 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:36:26.737 DEBUG:boto:Data: 2024-03-04T16:36:26.737 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.737 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.737 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.737 DEBUG:boto:Params: {} 2024-03-04T16:36:26.738 DEBUG:boto:Token: None 2024-03-04T16:36:26.738 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-25/ 2024-03-04T16:36:26.738 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pFUKvv8XtfCzK2FGsSaRKXc+5Ag= 2024-03-04T16:36:26.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pFUKvv8XtfCzK2FGsSaRKXc+5Ag=', 'Content-Length': '0'} 2024-03-04T16:36:26.763 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f809eb86c667fa95-0065e5f88a-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.763 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-26 2024-03-04T16:36:26.764 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:36:26.764 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:36:26.764 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.764 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:36:26.764 DEBUG:boto:Data: 2024-03-04T16:36:26.765 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.765 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.765 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.765 DEBUG:boto:Params: {} 2024-03-04T16:36:26.765 DEBUG:boto:Token: None 2024-03-04T16:36:26.766 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-26/ 2024-03-04T16:36:26.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Gi1UMK0KBgqyh7hm7mBdJyQ+rUw= 2024-03-04T16:36:26.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Gi1UMK0KBgqyh7hm7mBdJyQ+rUw=', 'Content-Length': '0'} 2024-03-04T16:36:26.789 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8134ecaf6ea67dd-0065e5f88a-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.789 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:36:26.790 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:36:26.790 DEBUG:boto:path=/sbnvju-24/?versioning 2024-03-04T16:36:26.790 DEBUG:boto:auth_path=/sbnvju-24/?versioning 2024-03-04T16:36:26.790 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.791 DEBUG:boto:Path: /sbnvju-24/?versioning 2024-03-04T16:36:26.791 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:36:26.791 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.791 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:26.791 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.791 DEBUG:boto:Params: {} 2024-03-04T16:36:26.792 DEBUG:boto:Token: None 2024-03-04T16:36:26.792 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-24/?versioning 2024-03-04T16:36:26.792 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:C6/JDloozU3PWVEttSZKnkVDGs4= 2024-03-04T16:36:26.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:C6/JDloozU3PWVEttSZKnkVDGs4=', 'Content-Length': '228'} 2024-03-04T16:36:26.798 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abcc121c16444d80-0065e5f88a-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.798 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:36:26.798 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:36:26.798 DEBUG:boto:path=/sbnvju-25/?versioning 2024-03-04T16:36:26.799 DEBUG:boto:auth_path=/sbnvju-25/?versioning 2024-03-04T16:36:26.799 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.799 DEBUG:boto:Path: /sbnvju-25/?versioning 2024-03-04T16:36:26.799 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:36:26.799 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.799 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:26.800 DEBUG:boto:Port: 8000 2024-03-04T16:36:26.800 DEBUG:boto:Params: {} 2024-03-04T16:36:26.800 DEBUG:boto:Token: None 2024-03-04T16:36:26.800 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-25/?versioning 2024-03-04T16:36:26.801 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:18mJ1ZTmBCSwNEyr+p+lD4LnxKA= 2024-03-04T16:36:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:18mJ1ZTmBCSwNEyr+p+lD4LnxKA=', 'Content-Length': '228'} 2024-03-04T16:36:26.821 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042f6de022ef87ddc-0065e5f88a-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.821 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:36:26.822 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:36:26.822 DEBUG:boto:path=/sbnvju-26/?versioning 2024-03-04T16:36:26.822 DEBUG:boto:auth_path=/sbnvju-26/?versioning 2024-03-04T16:36:26.822 DEBUG:boto:Method: PUT 2024-03-04T16:36:26.823 DEBUG:boto:Path: /sbnvju-26/?versioning 2024-03-04T16:36:26.823 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:36:26.823 DEBUG:boto:Headers: {} 2024-03-04T16:36:26.823 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:26.823 DEBUG:boto:Port: 8001 2024-03-04T16:36:26.823 DEBUG:boto:Params: {} 2024-03-04T16:36:26.824 DEBUG:boto:Token: None 2024-03-04T16:36:26.824 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:36:26 GMT /sbnvju-26/?versioning 2024-03-04T16:36:26.824 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mThONLYOp4gELMW69jnOAAi4I4A= 2024-03-04T16:36:26.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:26 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mThONLYOp4gELMW69jnOAAi4I4A=', 'Content-Length': '228'} 2024-03-04T16:36:26.832 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b12be421ac6fd91a-0065e5f88a-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:26 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:26.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:26.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:26.833 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:36:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:36:26.947 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570186.841908_341.1","last_update":"2024-03-04T16:36:26.841908Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:36:26.947 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:26.948 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:36:26.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:26.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:26.948 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:36:27.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:27.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:27.052 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:27.063+0000 7f04619af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:27.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:27.064+0000 7f04619af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:27.073 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.841908_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.841908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:27Z"} 2024-03-04T16:36:27.073 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.841908_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.841908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:27Z"} 2024-03-04T16:36:27.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570186.841908_341.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:36:26.841908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:36:27.074 DEBUG:rgw_multi.tests:log_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:27.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:27.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:36:27.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:36:27.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:27.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:27.074 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:36:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:27.177+0000 7f32ebe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:27.178+0000 7f32ebe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:27.185 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.789555_334.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.789555Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:27Z"} 2024-03-04T16:36:27.186 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.789555_334.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.789555Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:27Z"} 2024-03-04T16:36:27.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570186.789555_334.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:36:26.789555Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:36:27.186 DEBUG:rgw_multi.tests:log_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:27.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570186.789555_334.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:27.186 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1709570186.841908_341.1 target=1_1709570186.789555_334.1 2024-03-04T16:36:32.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:32.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:32.187 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:36:32.241 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:32.241 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:32.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:32.302+0000 7efc02467800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:32.303+0000 7efc02467800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:32.311 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.742749_325.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.742749Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:32Z"} 2024-03-04T16:36:32.312 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.742749_325.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.742749Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:32Z"} 2024-03-04T16:36:32.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570186.742749_325.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:36:26.742749Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:36:32.312 DEBUG:rgw_multi.tests:log_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:32.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570186.742749_325.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:32.312 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1709570186.841908_341.1 target=1_1709570186.742749_325.1 2024-03-04T16:36:37.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:37.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:36:37.314 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:36:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:37.430+0000 7f8313b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:36:37.431+0000 7f8313b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:36:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.841908_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.841908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:37Z"} 2024-03-04T16:36:37.441 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570186.841908_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:36:26.841908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:36:37Z"} 2024-03-04T16:36:37.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570186.841908_341.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:36:26.841908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:36:37.441 DEBUG:rgw_multi.tests:log_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:37.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570186.841908_341.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:36:37.441 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:36:37.441 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:36:37.442 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:36:37.442 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.442 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:36:37.442 DEBUG:boto:Data: 2024-03-04T16:36:37.442 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.442 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:37.442 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.442 DEBUG:boto:Params: {} 2024-03-04T16:36:37.442 DEBUG:boto:Token: None 2024-03-04T16:36:37.443 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-24/ 2024-03-04T16:36:37.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Of+1c/n1OedJsXlpZfhIfF/XAHg= 2024-03-04T16:36:37.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Of+1c/n1OedJsXlpZfhIfF/XAHg=', 'Content-Length': '0'} 2024-03-04T16:36:37.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ece1eab814859f9-0065e5f895-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.446 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:36:37.446 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:36:37.446 DEBUG:boto:Method: PUT 2024-03-04T16:36:37.446 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:36:37.446 DEBUG:boto:Data: 2024-03-04T16:36:37.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:36:37.446 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:37.446 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.447 DEBUG:boto:Params: {} 2024-03-04T16:36:37.447 DEBUG:boto:Token: None 2024-03-04T16:36:37.447 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-24/obj 2024-03-04T16:36:37.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ap4ApS5cvkI+2go9b8le+IlEb60= 2024-03-04T16:36:37.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ap4ApS5cvkI+2go9b8le+IlEb60='} 2024-03-04T16:36:37.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-amz-request-id', 'tx000005af30016234aede4-0065e5f895-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.493 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:36:37.493 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:36:37.493 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.493 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:36:37.493 DEBUG:boto:Data: 2024-03-04T16:36:37.493 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.493 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:37.493 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.493 DEBUG:boto:Params: {} 2024-03-04T16:36:37.493 DEBUG:boto:Token: None 2024-03-04T16:36:37.494 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-24/ 2024-03-04T16:36:37.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Of+1c/n1OedJsXlpZfhIfF/XAHg= 2024-03-04T16:36:37.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Of+1c/n1OedJsXlpZfhIfF/XAHg=', 'Content-Length': '0'} 2024-03-04T16:36:37.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc593c0dbb89a67f-0065e5f895-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.496 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:36:37.497 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:36:37.497 DEBUG:boto:Method: DELETE 2024-03-04T16:36:37.497 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:36:37.497 DEBUG:boto:Data: 2024-03-04T16:36:37.497 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:36:37.497 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.497 DEBUG:boto:Params: {} 2024-03-04T16:36:37.497 DEBUG:boto:Token: None 2024-03-04T16:36:37.497 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-24/obj 2024-03-04T16:36:37.498 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ajyMN6sRz/dV9TzBVjL34FCXa50= 2024-03-04T16:36:37.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ajyMN6sRz/dV9TzBVjL34FCXa50=', 'Content-Length': '0'} 2024-03-04T16:36:37.504 DEBUG:boto:Response headers: [('x-amz-version-id', 'G6roDmNOlkMmyGRmpTrHH741MwV0jsl'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000057ef67f16d66011a-0065e5f895-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.504 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:36:37.505 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:36:37.505 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.505 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:36:37.505 DEBUG:boto:Data: 2024-03-04T16:36:37.505 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.505 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:37.505 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.505 DEBUG:boto:Params: {} 2024-03-04T16:36:37.505 DEBUG:boto:Token: None 2024-03-04T16:36:37.505 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-25/ 2024-03-04T16:36:37.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6jUkVHn0GJp9v1dYqe6as4kC/bM= 2024-03-04T16:36:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6jUkVHn0GJp9v1dYqe6as4kC/bM=', 'Content-Length': '0'} 2024-03-04T16:36:37.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dfdbe3203acf0dd-0065e5f895-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.508 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:36:37.509 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:36:37.509 DEBUG:boto:Method: PUT 2024-03-04T16:36:37.509 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:36:37.509 DEBUG:boto:Data: 2024-03-04T16:36:37.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:36:37.509 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:37.509 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.509 DEBUG:boto:Params: {} 2024-03-04T16:36:37.509 DEBUG:boto:Token: None 2024-03-04T16:36:37.509 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-25/obj 2024-03-04T16:36:37.510 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:yRYHMQhrB2fAtRdbbJZNgu3PsJ4= 2024-03-04T16:36:37.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yRYHMQhrB2fAtRdbbJZNgu3PsJ4='} 2024-03-04T16:36:37.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-amz-request-id', 'tx000009d76b2dec1da3e85-0065e5f895-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.552 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:36:37.553 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:36:37.553 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.553 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:36:37.553 DEBUG:boto:Data: 2024-03-04T16:36:37.553 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.553 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:37.553 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.553 DEBUG:boto:Params: {} 2024-03-04T16:36:37.553 DEBUG:boto:Token: None 2024-03-04T16:36:37.554 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-25/ 2024-03-04T16:36:37.554 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6jUkVHn0GJp9v1dYqe6as4kC/bM= 2024-03-04T16:36:37.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6jUkVHn0GJp9v1dYqe6as4kC/bM=', 'Content-Length': '0'} 2024-03-04T16:36:37.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fec61ecf070cedf3-0065e5f895-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.557 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:36:37.557 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:36:37.557 DEBUG:boto:Method: DELETE 2024-03-04T16:36:37.557 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:36:37.557 DEBUG:boto:Data: 2024-03-04T16:36:37.557 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.557 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:36:37.557 DEBUG:boto:Port: 8000 2024-03-04T16:36:37.558 DEBUG:boto:Params: {} 2024-03-04T16:36:37.558 DEBUG:boto:Token: None 2024-03-04T16:36:37.558 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-25/obj 2024-03-04T16:36:37.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H2XEFdOVYeZKfWRBh84VSaMoHKk= 2024-03-04T16:36:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H2XEFdOVYeZKfWRBh84VSaMoHKk=', 'Content-Length': '0'} 2024-03-04T16:36:37.566 DEBUG:boto:Response headers: [('x-amz-version-id', 'fpwbOan4pTZ3sglQ4vZBwDaFvwwq-RF'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000ffd0180a6e782b11-0065e5f895-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.567 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:36:37.567 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:36:37.567 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.567 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:36:37.567 DEBUG:boto:Data: 2024-03-04T16:36:37.567 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.567 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:37.567 DEBUG:boto:Port: 8001 2024-03-04T16:36:37.567 DEBUG:boto:Params: {} 2024-03-04T16:36:37.568 DEBUG:boto:Token: None 2024-03-04T16:36:37.568 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-26/ 2024-03-04T16:36:37.568 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3UVsHwEKpE7/iNXxPvfuziZifoM= 2024-03-04T16:36:37.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3UVsHwEKpE7/iNXxPvfuziZifoM=', 'Content-Length': '0'} 2024-03-04T16:36:37.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cff6eccf78b4a6d-0065e5f895-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.570 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:36:37.571 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:36:37.571 DEBUG:boto:Method: PUT 2024-03-04T16:36:37.571 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:36:37.571 DEBUG:boto:Data: 2024-03-04T16:36:37.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:36:37.571 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:37.571 DEBUG:boto:Port: 8001 2024-03-04T16:36:37.571 DEBUG:boto:Params: {} 2024-03-04T16:36:37.571 DEBUG:boto:Token: None 2024-03-04T16:36:37.571 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-26/obj 2024-03-04T16:36:37.572 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:udpc7ncr53Wx7GFt2CfLkHSuu34= 2024-03-04T16:36:37.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:udpc7ncr53Wx7GFt2CfLkHSuu34='} 2024-03-04T16:36:37.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-amz-request-id', 'tx00000038071a8bd4a5972-0065e5f895-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.582 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:36:37.582 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:36:37.583 DEBUG:boto:Method: HEAD 2024-03-04T16:36:37.583 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:36:37.583 DEBUG:boto:Data: 2024-03-04T16:36:37.583 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.583 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:37.583 DEBUG:boto:Port: 8001 2024-03-04T16:36:37.583 DEBUG:boto:Params: {} 2024-03-04T16:36:37.583 DEBUG:boto:Token: None 2024-03-04T16:36:37.583 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-26/ 2024-03-04T16:36:37.583 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3UVsHwEKpE7/iNXxPvfuziZifoM= 2024-03-04T16:36:37.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3UVsHwEKpE7/iNXxPvfuziZifoM=', 'Content-Length': '0'} 2024-03-04T16:36:37.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee1353a9b569b1d5-0065e5f895-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.586 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:36:37.586 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:36:37.586 DEBUG:boto:Method: DELETE 2024-03-04T16:36:37.586 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:36:37.586 DEBUG:boto:Data: 2024-03-04T16:36:37.586 DEBUG:boto:Headers: {} 2024-03-04T16:36:37.586 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:36:37.586 DEBUG:boto:Port: 8001 2024-03-04T16:36:37.586 DEBUG:boto:Params: {} 2024-03-04T16:36:37.586 DEBUG:boto:Token: None 2024-03-04T16:36:37.587 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:36:37 GMT /sbnvju-26/obj 2024-03-04T16:36:37.587 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wTvFz/M4zRbm7tjrekqjPc+vyXY= 2024-03-04T16:36:37.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:36:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wTvFz/M4zRbm7tjrekqjPc+vyXY=', 'Content-Length': '0'} 2024-03-04T16:36:37.593 DEBUG:boto:Response headers: [('x-amz-version-id', 'kSy06mbdT5st3Ao6UR-UOYGVJSFhd2r'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000008bc48cc57adc1c7f-0065e5f895-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:36:37.593 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-24 2024-03-04T16:36:37.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:37.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:36:37.594 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:36:37.654 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:36:37.654 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:36:37.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:37.718+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:36:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:42.718+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:36:47.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:47.719+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:36:52.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:52.721+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:36:57.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:36:57.720+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:37:02.710 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:02.721+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:37:07.710 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:07.722+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:37:12.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:12.723+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:37:17.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:17.723+0000 7f9a29da2800 1 waiting to reach incremental sync.. 2024-03-04T16:37:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:22.725+0000 7f9a29da2800 1 bucket sync caught up with source: 2024-03-04T16:37:22.713 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.47.12, , ] 2024-03-04T16:37:22.713 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.47.12, , ] 2024-03-04T16:37:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:22.725+0000 7f9a29da2800 0 bucket checkpoint complete 2024-03-04T16:37:22.737 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-24 2024-03-04T16:37:22.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:22.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:22.737 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:22.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:22.867+0000 7f0f5c69d800 1 bucket sync caught up with source: 2024-03-04T16:37:22.856 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.47.12, , ] 2024-03-04T16:37:22.856 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.47.12, , ] 2024-03-04T16:37:22.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:22.867+0000 7f0f5c69d800 0 bucket checkpoint complete 2024-03-04T16:37:22.875 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-24 2024-03-04T16:37:22.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:22.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:22.875 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:22.930 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:22.930 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:22.996+0000 7f7eb2a67800 1 bucket sync caught up with source: 2024-03-04T16:37:22.984 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.38.12, , ] 2024-03-04T16:37:22.984 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.38.12, , ] 2024-03-04T16:37:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:22.996+0000 7f7eb2a67800 0 bucket checkpoint complete 2024-03-04T16:37:23.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-24 2024-03-04T16:37:23.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.003 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.057 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.057 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.121+0000 7f2bbba2e800 1 bucket sync caught up with source: 2024-03-04T16:37:23.109 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.38.12, , ] 2024-03-04T16:37:23.109 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.38.12, , ] 2024-03-04T16:37:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.121+0000 7f2bbba2e800 0 bucket checkpoint complete 2024-03-04T16:37:23.127 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-24 2024-03-04T16:37:23.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.128 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:23.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.246+0000 7f42fd133800 1 bucket sync caught up with source: 2024-03-04T16:37:23.234 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.56.12, , ] 2024-03-04T16:37:23.234 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.56.12, , ] 2024-03-04T16:37:23.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.246+0000 7f42fd133800 0 bucket checkpoint complete 2024-03-04T16:37:23.253 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-24 2024-03-04T16:37:23.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.253 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:23.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:23.378+0000 7f428d636800 1 bucket sync caught up with source: 2024-03-04T16:37:23.366 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.56.12, , ] 2024-03-04T16:37:23.366 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.56.12, , ] 2024-03-04T16:37:23.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:23.378+0000 7f428d636800 0 bucket checkpoint complete 2024-03-04T16:37:23.385 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-24 zones={test-zone2, test-zone1} 2024-03-04T16:37:23.385 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.385 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.386 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.386 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.386 DEBUG:boto:Data: 2024-03-04T16:37:23.386 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.386 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.386 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.386 DEBUG:boto:Params: {} 2024-03-04T16:37:23.386 DEBUG:boto:Token: None 2024-03-04T16:37:23.387 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.387 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fc20a8db97f8256-0065e5f8c3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.390 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.390 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.390 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.390 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.390 DEBUG:boto:Data: 2024-03-04T16:37:23.391 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.391 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.391 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.391 DEBUG:boto:Params: {} 2024-03-04T16:37:23.391 DEBUG:boto:Token: None 2024-03-04T16:37:23.391 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.391 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066160a2ac99460bd-0065e5f8c3-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.394 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:23.394 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.394 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.394 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.394 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.395 DEBUG:boto:Method: GET 2024-03-04T16:37:23.395 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.395 DEBUG:boto:Data: 2024-03-04T16:37:23.395 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.395 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.395 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.395 DEBUG:boto:Params: {} 2024-03-04T16:37:23.395 DEBUG:boto:Token: None 2024-03-04T16:37:23.395 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.395 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b658d1bca9031fdd-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.398 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.399 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:23.399 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.399 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.400 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.400 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.400 DEBUG:boto:Method: GET 2024-03-04T16:37:23.400 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.400 DEBUG:boto:Data: 2024-03-04T16:37:23.400 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.400 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.400 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.400 DEBUG:boto:Params: {} 2024-03-04T16:37:23.400 DEBUG:boto:Token: None 2024-03-04T16:37:23.400 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.401 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0472df1612f710-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.403 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.404 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.404 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.404 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.404 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.404 DEBUG:boto:Method: GET 2024-03-04T16:37:23.404 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.404 DEBUG:boto:Data: 2024-03-04T16:37:23.404 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.404 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.404 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.405 DEBUG:boto:Params: {} 2024-03-04T16:37:23.405 DEBUG:boto:Token: None 2024-03-04T16:37:23.405 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.405 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096dda6b48862f3f7-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.407 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.408 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.408 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.408 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.408 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.408 DEBUG:boto:Method: GET 2024-03-04T16:37:23.408 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.408 DEBUG:boto:Data: 2024-03-04T16:37:23.408 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.408 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.409 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.409 DEBUG:boto:Params: {} 2024-03-04T16:37:23.409 DEBUG:boto:Token: None 2024-03-04T16:37:23.409 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093302f54b4289017-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.412 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.412 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.412 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.413 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.413 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.413 DEBUG:boto:Data: 2024-03-04T16:37:23.413 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.413 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.413 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.413 DEBUG:boto:Params: {} 2024-03-04T16:37:23.413 DEBUG:boto:Token: None 2024-03-04T16:37:23.413 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.414 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.415 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004d27ae0e1b8acfc8-0065e5f8c3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.415 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.416 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.416 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.416 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.416 DEBUG:boto:Data: 2024-03-04T16:37:23.416 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.416 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.416 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.416 DEBUG:boto:Params: {} 2024-03-04T16:37:23.416 DEBUG:boto:Token: None 2024-03-04T16:37:23.416 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.417 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.418 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000082827d36ffcda84f-0065e5f8c3-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.419 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.419 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.419 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.419 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.419 DEBUG:boto:Method: GET 2024-03-04T16:37:23.419 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.419 DEBUG:boto:Data: 2024-03-04T16:37:23.419 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.419 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.420 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.420 DEBUG:boto:Params: {} 2024-03-04T16:37:23.420 DEBUG:boto:Token: None 2024-03-04T16:37:23.420 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.420 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040641452d3555ffd-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.422 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.422 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.422 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.422 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.422 DEBUG:boto:Method: GET 2024-03-04T16:37:23.422 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.423 DEBUG:boto:Data: 2024-03-04T16:37:23.423 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.423 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.423 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.423 DEBUG:boto:Params: {} 2024-03-04T16:37:23.423 DEBUG:boto:Token: None 2024-03-04T16:37:23.423 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.423 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000690b550629ca59b9-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.425 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.425 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.425 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.425 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.425 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.425 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.425 DEBUG:boto:Data: 2024-03-04T16:37:23.426 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.426 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.426 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.426 DEBUG:boto:Params: {} 2024-03-04T16:37:23.426 DEBUG:boto:Token: None 2024-03-04T16:37:23.426 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.426 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1cee4320496f426-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.428 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.428 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.428 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.428 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.429 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.429 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.429 DEBUG:boto:Data: 2024-03-04T16:37:23.429 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.429 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.429 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.429 DEBUG:boto:Params: {} 2024-03-04T16:37:23.429 DEBUG:boto:Token: None 2024-03-04T16:37:23.429 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.429 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a01ebefc84a11beb-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.432 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.432 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.432 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.432 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.432 DEBUG:boto:Method: GET 2024-03-04T16:37:23.432 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.432 DEBUG:boto:Data: 2024-03-04T16:37:23.432 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.433 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.433 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.433 DEBUG:boto:Params: {} 2024-03-04T16:37:23.433 DEBUG:boto:Token: None 2024-03-04T16:37:23.433 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.433 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081fed62b7b447bf5-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.435 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.435 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.435 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.435 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.435 DEBUG:boto:Method: GET 2024-03-04T16:37:23.435 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.435 DEBUG:boto:Data: 2024-03-04T16:37:23.436 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.436 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.436 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.436 DEBUG:boto:Params: {} 2024-03-04T16:37:23.436 DEBUG:boto:Token: None 2024-03-04T16:37:23.436 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.436 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef0cc261212b9a60-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.438 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.438 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.438 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.438 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.439 DEBUG:boto:Method: GET 2024-03-04T16:37:23.439 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.439 DEBUG:boto:Data: 2024-03-04T16:37:23.439 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.439 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.439 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.439 DEBUG:boto:Params: {} 2024-03-04T16:37:23.439 DEBUG:boto:Token: None 2024-03-04T16:37:23.439 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.439 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208ce659029ac569-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.442 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.442 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.442 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.442 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.442 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.443 DEBUG:boto:Method: GET 2024-03-04T16:37:23.443 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.443 DEBUG:boto:Data: 2024-03-04T16:37:23.443 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.443 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.443 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.443 DEBUG:boto:Params: {} 2024-03-04T16:37:23.443 DEBUG:boto:Token: None 2024-03-04T16:37:23.443 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.443 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f73e06fe358610cf-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.492 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.492 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.492 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.492 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.493 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.493 DEBUG:boto:Data: 2024-03-04T16:37:23.493 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.493 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.493 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.493 DEBUG:boto:Params: {} 2024-03-04T16:37:23.493 DEBUG:boto:Token: None 2024-03-04T16:37:23.493 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.494 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.496 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001f995eaa9dc7407e-0065e5f8c3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.496 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.496 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.496 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.496 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.496 DEBUG:boto:Data: 2024-03-04T16:37:23.497 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.497 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.497 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.497 DEBUG:boto:Params: {} 2024-03-04T16:37:23.497 DEBUG:boto:Token: None 2024-03-04T16:37:23.497 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.499 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008af4004206ac3326-0065e5f8c3-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.499 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-24 zones={test-zone2, test-zone1} 2024-03-04T16:37:23.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-24 zones={test-zone3, test-zone1} 2024-03-04T16:37:23.500 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.500 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.500 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.500 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.500 DEBUG:boto:Data: 2024-03-04T16:37:23.500 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.500 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.500 DEBUG:boto:Params: {} 2024-03-04T16:37:23.501 DEBUG:boto:Token: None 2024-03-04T16:37:23.501 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.501 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2044d6848ddef8f-0065e5f8c3-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.503 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.503 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.503 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.503 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.504 DEBUG:boto:Data: 2024-03-04T16:37:23.504 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.504 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.504 DEBUG:boto:Params: {} 2024-03-04T16:37:23.504 DEBUG:boto:Token: None 2024-03-04T16:37:23.504 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.504 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ea597bd388446c6-0065e5f8c3-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.507 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:23.507 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.507 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.507 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.507 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.507 DEBUG:boto:Method: GET 2024-03-04T16:37:23.507 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.507 DEBUG:boto:Data: 2024-03-04T16:37:23.508 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.508 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.508 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.508 DEBUG:boto:Params: {} 2024-03-04T16:37:23.508 DEBUG:boto:Token: None 2024-03-04T16:37:23.508 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052290cd0c16e0249-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.510 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:23.511 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.511 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.511 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.511 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.512 DEBUG:boto:Method: GET 2024-03-04T16:37:23.512 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.512 DEBUG:boto:Data: 2024-03-04T16:37:23.512 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.512 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.512 DEBUG:boto:Params: {} 2024-03-04T16:37:23.512 DEBUG:boto:Token: None 2024-03-04T16:37:23.512 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.512 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a3f2b474984175-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.515 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.515 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.515 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.515 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.515 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.516 DEBUG:boto:Method: GET 2024-03-04T16:37:23.516 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.516 DEBUG:boto:Data: 2024-03-04T16:37:23.516 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.516 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.516 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.516 DEBUG:boto:Params: {} 2024-03-04T16:37:23.516 DEBUG:boto:Token: None 2024-03-04T16:37:23.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.516 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccc7be3d683d7df-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.519 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.519 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.519 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.519 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.519 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.519 DEBUG:boto:Method: GET 2024-03-04T16:37:23.519 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.520 DEBUG:boto:Data: 2024-03-04T16:37:23.520 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.520 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.520 DEBUG:boto:Params: {} 2024-03-04T16:37:23.520 DEBUG:boto:Token: None 2024-03-04T16:37:23.520 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.520 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000640105b7ee3a3c5c-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.522 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.523 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.523 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.523 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.523 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.523 DEBUG:boto:Data: 2024-03-04T16:37:23.523 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.523 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.523 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.524 DEBUG:boto:Params: {} 2024-03-04T16:37:23.524 DEBUG:boto:Token: None 2024-03-04T16:37:23.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.526 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000945a4eab185ea224-0065e5f8c3-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.526 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.526 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.526 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.526 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.526 DEBUG:boto:Data: 2024-03-04T16:37:23.527 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.527 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.527 DEBUG:boto:Params: {} 2024-03-04T16:37:23.527 DEBUG:boto:Token: None 2024-03-04T16:37:23.527 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.529 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000882443c0e929affa-0065e5f8c3-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.529 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.530 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.530 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.530 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.530 DEBUG:boto:Method: GET 2024-03-04T16:37:23.530 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.530 DEBUG:boto:Data: 2024-03-04T16:37:23.530 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.530 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.530 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.530 DEBUG:boto:Params: {} 2024-03-04T16:37:23.530 DEBUG:boto:Token: None 2024-03-04T16:37:23.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.531 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028c90725a8c4ee78-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.533 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.533 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.533 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.533 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.533 DEBUG:boto:Method: GET 2024-03-04T16:37:23.533 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.533 DEBUG:boto:Data: 2024-03-04T16:37:23.533 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.533 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.534 DEBUG:boto:Params: {} 2024-03-04T16:37:23.534 DEBUG:boto:Token: None 2024-03-04T16:37:23.534 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.534 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007dbdb482ec370016-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.536 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.536 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.536 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.536 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.536 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.536 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.536 DEBUG:boto:Data: 2024-03-04T16:37:23.536 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.536 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.536 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.537 DEBUG:boto:Params: {} 2024-03-04T16:37:23.537 DEBUG:boto:Token: None 2024-03-04T16:37:23.537 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.537 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a42ad89161390c6-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.539 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.539 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.539 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.539 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.539 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.539 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.539 DEBUG:boto:Data: 2024-03-04T16:37:23.539 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.539 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.539 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.539 DEBUG:boto:Params: {} 2024-03-04T16:37:23.540 DEBUG:boto:Token: None 2024-03-04T16:37:23.540 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.540 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6f6cd188e4d8b08-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.542 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.542 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.542 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.542 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.542 DEBUG:boto:Method: GET 2024-03-04T16:37:23.542 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.542 DEBUG:boto:Data: 2024-03-04T16:37:23.542 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.542 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.542 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.542 DEBUG:boto:Params: {} 2024-03-04T16:37:23.543 DEBUG:boto:Token: None 2024-03-04T16:37:23.543 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c55b02949ea41759-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.545 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.545 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.545 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.545 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.545 DEBUG:boto:Method: GET 2024-03-04T16:37:23.545 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.545 DEBUG:boto:Data: 2024-03-04T16:37:23.545 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.545 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.546 DEBUG:boto:Params: {} 2024-03-04T16:37:23.546 DEBUG:boto:Token: None 2024-03-04T16:37:23.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007323e634272ee560-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.548 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.548 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.548 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.548 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.548 DEBUG:boto:Method: GET 2024-03-04T16:37:23.548 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.548 DEBUG:boto:Data: 2024-03-04T16:37:23.549 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.549 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.549 DEBUG:boto:Params: {} 2024-03-04T16:37:23.549 DEBUG:boto:Token: None 2024-03-04T16:37:23.549 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf82e65d57bb4128-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.552 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.552 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.552 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.552 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.552 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.552 DEBUG:boto:Method: GET 2024-03-04T16:37:23.552 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.552 DEBUG:boto:Data: 2024-03-04T16:37:23.552 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.553 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.553 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.553 DEBUG:boto:Params: {} 2024-03-04T16:37:23.553 DEBUG:boto:Token: None 2024-03-04T16:37:23.553 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098160117b8ee007b-0065e5f8c3-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.555 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.555 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.556 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.556 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.556 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.556 DEBUG:boto:Data: 2024-03-04T16:37:23.556 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.556 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.556 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.556 DEBUG:boto:Params: {} 2024-03-04T16:37:23.556 DEBUG:boto:Token: None 2024-03-04T16:37:23.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.557 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.559 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000017d439bffce64b97-0065e5f8c3-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.559 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.559 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.559 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.559 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.559 DEBUG:boto:Data: 2024-03-04T16:37:23.559 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.560 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:23.560 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.560 DEBUG:boto:Params: {} 2024-03-04T16:37:23.560 DEBUG:boto:Token: None 2024-03-04T16:37:23.560 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.560 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.562 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002037d4bae1ff91c0-0065e5f8c3-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.563 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-24 zones={test-zone3, test-zone1} 2024-03-04T16:37:23.563 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-24 zones={test-zone3, test-zone2} 2024-03-04T16:37:23.563 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.563 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.563 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.563 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.563 DEBUG:boto:Data: 2024-03-04T16:37:23.563 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.563 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.563 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.563 DEBUG:boto:Params: {} 2024-03-04T16:37:23.563 DEBUG:boto:Token: None 2024-03-04T16:37:23.564 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.564 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097ceef97c66e7132-0065e5f8c3-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.567 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.567 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.567 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.567 DEBUG:boto:Path: /sbnvju-24/ 2024-03-04T16:37:23.567 DEBUG:boto:Data: 2024-03-04T16:37:23.567 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.567 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.567 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.567 DEBUG:boto:Params: {} 2024-03-04T16:37:23.567 DEBUG:boto:Token: None 2024-03-04T16:37:23.568 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.568 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM= 2024-03-04T16:37:23.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oU+ebK0IPvUyoNXLeKOcHzExmqM=', 'Content-Length': '0'} 2024-03-04T16:37:23.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006336721e8a244653-0065e5f8c3-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.570 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:23.570 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.570 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.570 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.570 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.570 DEBUG:boto:Method: GET 2024-03-04T16:37:23.571 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.571 DEBUG:boto:Data: 2024-03-04T16:37:23.571 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.571 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.571 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.571 DEBUG:boto:Params: {} 2024-03-04T16:37:23.571 DEBUG:boto:Token: None 2024-03-04T16:37:23.571 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c66a1b6fa6d2bbe-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.574 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:23.575 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.575 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.575 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.575 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.575 DEBUG:boto:Method: GET 2024-03-04T16:37:23.575 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.575 DEBUG:boto:Data: 2024-03-04T16:37:23.575 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.575 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.575 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.576 DEBUG:boto:Params: {} 2024-03-04T16:37:23.576 DEBUG:boto:Token: None 2024-03-04T16:37:23.576 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.576 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d9a91f26791d71-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.578 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:23.579 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.579 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.579 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.579 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.579 DEBUG:boto:Method: GET 2024-03-04T16:37:23.579 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.579 DEBUG:boto:Data: 2024-03-04T16:37:23.580 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.580 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.580 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.580 DEBUG:boto:Params: {} 2024-03-04T16:37:23.580 DEBUG:boto:Token: None 2024-03-04T16:37:23.580 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.580 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d1ef79e63e03b1-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.582 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.583 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.583 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.583 DEBUG:boto:path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.583 DEBUG:boto:auth_path=/sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.583 DEBUG:boto:Method: GET 2024-03-04T16:37:23.583 DEBUG:boto:Path: /sbnvju-24/?versions&max-keys=999 2024-03-04T16:37:23.583 DEBUG:boto:Data: 2024-03-04T16:37:23.583 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.583 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.584 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.584 DEBUG:boto:Params: {} 2024-03-04T16:37:23.584 DEBUG:boto:Token: None 2024-03-04T16:37:23.584 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/?versions 2024-03-04T16:37:23.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws= 2024-03-04T16:37:23.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pS0kjroArzJCQQ6B+x6DQM/Jzws=', 'Content-Length': '0'} 2024-03-04T16:37:23.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b162df9cdf04bd86-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.587 DEBUG:boto:b'sbnvju-24999falseobjG6roDmNOlkMmyGRmpTrHH741MwV0jsltrue2024-03-04T16:36:37.512Zrgw-multisite-test-userMultisite Test UserNormalobjZy2pa-jV9hZPhzR4imgY3hcqKo4G8sWfalse2024-03-04T16:36:37.461Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:23.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.587 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.587 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.587 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.587 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.587 DEBUG:boto:Data: 2024-03-04T16:37:23.588 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.588 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.588 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.588 DEBUG:boto:Params: {} 2024-03-04T16:37:23.588 DEBUG:boto:Token: None 2024-03-04T16:37:23.588 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.588 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.590 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fb6baed6479c2188-0065e5f8c3-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.590 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.590 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.590 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.590 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.590 DEBUG:boto:Data: 2024-03-04T16:37:23.591 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.591 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.591 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.591 DEBUG:boto:Params: {} 2024-03-04T16:37:23.591 DEBUG:boto:Token: None 2024-03-04T16:37:23.591 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.591 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.593 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d29e6e8e74624938-0065e5f8c3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.593 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.593 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.594 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.594 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.594 DEBUG:boto:Method: GET 2024-03-04T16:37:23.594 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.594 DEBUG:boto:Data: 2024-03-04T16:37:23.594 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.594 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.594 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.594 DEBUG:boto:Params: {} 2024-03-04T16:37:23.594 DEBUG:boto:Token: None 2024-03-04T16:37:23.595 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.595 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000524d3da750e1e3ef-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.597 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.597 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.597 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.597 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.597 DEBUG:boto:Method: GET 2024-03-04T16:37:23.597 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.597 DEBUG:boto:Data: 2024-03-04T16:37:23.597 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.597 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.597 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.598 DEBUG:boto:Params: {} 2024-03-04T16:37:23.598 DEBUG:boto:Token: None 2024-03-04T16:37:23.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000889b4954736bc797-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.600 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.600 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.600 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.600 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.600 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.601 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.601 DEBUG:boto:Data: 2024-03-04T16:37:23.601 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.601 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.601 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.601 DEBUG:boto:Params: {} 2024-03-04T16:37:23.601 DEBUG:boto:Token: None 2024-03-04T16:37:23.601 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000381b3203a2753e01-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.603 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.603 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.603 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.603 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.603 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.603 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.603 DEBUG:boto:Data: 2024-03-04T16:37:23.604 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.604 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.604 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.604 DEBUG:boto:Params: {} 2024-03-04T16:37:23.604 DEBUG:boto:Token: None 2024-03-04T16:37:23.604 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.604 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo= 2024-03-04T16:37:23.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2gDWOyazzGvCvIR7ZtVgueRkVRo=', 'Content-Length': '0'} 2024-03-04T16:37:23.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b385cc5fdc44f915-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:23.606 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.606 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.606 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.607 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.607 DEBUG:boto:Method: GET 2024-03-04T16:37:23.607 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.607 DEBUG:boto:Data: 2024-03-04T16:37:23.607 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.607 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.607 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.607 DEBUG:boto:Params: {} 2024-03-04T16:37:23.607 DEBUG:boto:Token: None 2024-03-04T16:37:23.607 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.608 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7ef95e604f97477-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.610 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.610 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.610 DEBUG:boto:path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.610 DEBUG:boto:auth_path=/sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.610 DEBUG:boto:Method: GET 2024-03-04T16:37:23.610 DEBUG:boto:Path: /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.610 DEBUG:boto:Data: 2024-03-04T16:37:23.610 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.610 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.610 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.611 DEBUG:boto:Params: {} 2024-03-04T16:37:23.611 DEBUG:boto:Token: None 2024-03-04T16:37:23.611 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj?versionId=Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW 2024-03-04T16:37:23.611 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0= 2024-03-04T16:37:23.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ENI4dnuJ18XDyNEXQTM8el/SRq0=', 'Content-Length': '0'} 2024-03-04T16:37:23.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'Zy2pa-jV9hZPhzR4imgY3hcqKo4G8sW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'fc294220-5038-4f25-aeee-cac2c5838729:sbnvju-24:fc294220-5038-4f25-aeee-cac2c5838729.4341.33'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042bb498ea0c879bc-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.613 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.613 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.613 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.613 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.613 DEBUG:boto:Method: GET 2024-03-04T16:37:23.613 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.614 DEBUG:boto:Data: 2024-03-04T16:37:23.614 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.614 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.614 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.614 DEBUG:boto:Params: {} 2024-03-04T16:37:23.614 DEBUG:boto:Token: None 2024-03-04T16:37:23.614 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.614 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c8f105776ee3c3-0065e5f8c3-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.660 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.660 DEBUG:boto:path=/sbnvju-24/ 2024-03-04T16:37:23.660 DEBUG:boto:auth_path=/sbnvju-24/ 2024-03-04T16:37:23.660 DEBUG:boto:path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.661 DEBUG:boto:auth_path=/sbnvju-24/?prefix=obj 2024-03-04T16:37:23.661 DEBUG:boto:Method: GET 2024-03-04T16:37:23.661 DEBUG:boto:Path: /sbnvju-24/?prefix=obj 2024-03-04T16:37:23.661 DEBUG:boto:Data: 2024-03-04T16:37:23.661 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.661 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.661 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.661 DEBUG:boto:Params: {} 2024-03-04T16:37:23.661 DEBUG:boto:Token: None 2024-03-04T16:37:23.661 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/ 2024-03-04T16:37:23.662 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0= 2024-03-04T16:37:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TjYMNERSr19uSJ7djTPCKGsWSl0=', 'Content-Length': '0'} 2024-03-04T16:37:23.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be98cad147f6a5c0-0065e5f8c3-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.664 DEBUG:boto:b'sbnvju-24obj1000false' 2024-03-04T16:37:23.664 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.664 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.665 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.665 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.665 DEBUG:boto:Data: 2024-03-04T16:37:23.665 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:23.665 DEBUG:boto:Port: 8001 2024-03-04T16:37:23.665 DEBUG:boto:Params: {} 2024-03-04T16:37:23.665 DEBUG:boto:Token: None 2024-03-04T16:37:23.665 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.665 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.667 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fffff3bb79b93f6c-0065e5f8c3-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.667 DEBUG:boto:path=/sbnvju-24/obj 2024-03-04T16:37:23.667 DEBUG:boto:auth_path=/sbnvju-24/obj 2024-03-04T16:37:23.668 DEBUG:boto:Method: HEAD 2024-03-04T16:37:23.668 DEBUG:boto:Path: /sbnvju-24/obj 2024-03-04T16:37:23.668 DEBUG:boto:Data: 2024-03-04T16:37:23.668 DEBUG:boto:Headers: {} 2024-03-04T16:37:23.668 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:23.668 DEBUG:boto:Port: 8000 2024-03-04T16:37:23.668 DEBUG:boto:Params: {} 2024-03-04T16:37:23.668 DEBUG:boto:Token: None 2024-03-04T16:37:23.668 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:23 GMT /sbnvju-24/obj 2024-03-04T16:37:23.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4= 2024-03-04T16:37:23.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H1tLArEElVxwwl22HPiHbsZgiB4=', 'Content-Length': '0'} 2024-03-04T16:37:23.671 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005ae9ce706c3aa1de-0065e5f8c3-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:23.671 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-24 zones={test-zone3, test-zone2} 2024-03-04T16:37:23.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-25 2024-03-04T16:37:23.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.671 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:23.779 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:23.791+0000 7f82f3f11800 1 bucket sync caught up with source: 2024-03-04T16:37:23.779 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.28.12, , ] 2024-03-04T16:37:23.780 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.28.12, , ] 2024-03-04T16:37:23.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:23.791+0000 7f82f3f11800 0 bucket checkpoint complete 2024-03-04T16:37:23.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-25 2024-03-04T16:37:23.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.798 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.853 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.853 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.916+0000 7f0c8688c800 1 bucket sync caught up with source: 2024-03-04T16:37:23.904 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.28.12, , ] 2024-03-04T16:37:23.904 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.28.12, , ] 2024-03-04T16:37:23.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:23.916+0000 7f0c8688c800 0 bucket checkpoint complete 2024-03-04T16:37:23.922 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-25 2024-03-04T16:37:23.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:23.923 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.040+0000 7fe5c1424800 1 bucket sync caught up with source: 2024-03-04T16:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.12, , ] 2024-03-04T16:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.32.12, , ] 2024-03-04T16:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.040+0000 7fe5c1424800 0 bucket checkpoint complete 2024-03-04T16:37:24.047 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-25 2024-03-04T16:37:24.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.048 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.165+0000 7f8ebb038800 1 bucket sync caught up with source: 2024-03-04T16:37:24.153 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.32.12, , ] 2024-03-04T16:37:24.153 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.32.12, , ] 2024-03-04T16:37:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.165+0000 7f8ebb038800 0 bucket checkpoint complete 2024-03-04T16:37:24.172 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-25 2024-03-04T16:37:24.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.173 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.289+0000 7f5fcbf1f800 1 bucket sync caught up with source: 2024-03-04T16:37:24.277 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.50.12, , ] 2024-03-04T16:37:24.277 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.50.12, , ] 2024-03-04T16:37:24.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.289+0000 7f5fcbf1f800 0 bucket checkpoint complete 2024-03-04T16:37:24.296 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-25 2024-03-04T16:37:24.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.296 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:24.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:24.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:24.412+0000 7fbac4c2e800 1 bucket sync caught up with source: 2024-03-04T16:37:24.400 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.50.12, , ] 2024-03-04T16:37:24.400 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.50.12, , ] 2024-03-04T16:37:24.400 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:24.412+0000 7fbac4c2e800 0 bucket checkpoint complete 2024-03-04T16:37:24.418 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-25 zones={test-zone2, test-zone1} 2024-03-04T16:37:24.418 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.418 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.418 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.418 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.418 DEBUG:boto:Data: 2024-03-04T16:37:24.418 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.419 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.419 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.419 DEBUG:boto:Params: {} 2024-03-04T16:37:24.419 DEBUG:boto:Token: None 2024-03-04T16:37:24.419 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.419 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043662a3c43263512-0065e5f8c4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.422 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.422 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.423 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.423 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.423 DEBUG:boto:Data: 2024-03-04T16:37:24.423 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.423 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.423 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.423 DEBUG:boto:Params: {} 2024-03-04T16:37:24.423 DEBUG:boto:Token: None 2024-03-04T16:37:24.423 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.423 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068aa7eda0033ec9e-0065e5f8c4-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:24.426 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.426 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.427 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.427 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.427 DEBUG:boto:Method: GET 2024-03-04T16:37:24.427 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.427 DEBUG:boto:Data: 2024-03-04T16:37:24.427 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.427 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.427 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.427 DEBUG:boto:Params: {} 2024-03-04T16:37:24.427 DEBUG:boto:Token: None 2024-03-04T16:37:24.428 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.428 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f03401ea0e7805-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.430 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:24.431 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.431 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.431 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.431 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.432 DEBUG:boto:Method: GET 2024-03-04T16:37:24.432 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.432 DEBUG:boto:Data: 2024-03-04T16:37:24.432 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.432 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.432 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.432 DEBUG:boto:Params: {} 2024-03-04T16:37:24.432 DEBUG:boto:Token: None 2024-03-04T16:37:24.432 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.432 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073392cef908a5df-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.435 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.436 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.436 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.436 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.436 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.436 DEBUG:boto:Method: GET 2024-03-04T16:37:24.436 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.436 DEBUG:boto:Data: 2024-03-04T16:37:24.436 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.436 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.436 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.436 DEBUG:boto:Params: {} 2024-03-04T16:37:24.437 DEBUG:boto:Token: None 2024-03-04T16:37:24.437 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.437 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e395ef7d849889-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.439 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.440 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.440 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.440 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.440 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.440 DEBUG:boto:Method: GET 2024-03-04T16:37:24.440 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.440 DEBUG:boto:Data: 2024-03-04T16:37:24.440 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.440 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.440 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.441 DEBUG:boto:Params: {} 2024-03-04T16:37:24.441 DEBUG:boto:Token: None 2024-03-04T16:37:24.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028fcebf645dbc472-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.443 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.444 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.444 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.444 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.444 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.444 DEBUG:boto:Data: 2024-03-04T16:37:24.444 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.444 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.445 DEBUG:boto:Params: {} 2024-03-04T16:37:24.445 DEBUG:boto:Token: None 2024-03-04T16:37:24.445 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.445 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.447 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000070affdada1b0d87b-0065e5f8c4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.447 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.447 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.447 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.447 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.447 DEBUG:boto:Data: 2024-03-04T16:37:24.448 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.448 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.448 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.448 DEBUG:boto:Params: {} 2024-03-04T16:37:24.448 DEBUG:boto:Token: None 2024-03-04T16:37:24.448 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.448 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.450 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000346ebfd1542c6023-0065e5f8c4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.451 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.451 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.451 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.451 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.451 DEBUG:boto:Method: GET 2024-03-04T16:37:24.451 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.451 DEBUG:boto:Data: 2024-03-04T16:37:24.451 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.451 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.451 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.452 DEBUG:boto:Params: {} 2024-03-04T16:37:24.452 DEBUG:boto:Token: None 2024-03-04T16:37:24.452 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.452 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1a8d501ffd1001f-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.454 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.454 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.454 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.454 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.454 DEBUG:boto:Method: GET 2024-03-04T16:37:24.455 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.455 DEBUG:boto:Data: 2024-03-04T16:37:24.455 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.455 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.455 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.455 DEBUG:boto:Params: {} 2024-03-04T16:37:24.455 DEBUG:boto:Token: None 2024-03-04T16:37:24.455 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.455 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008eaeeb0440d5590d-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.457 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.458 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.458 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.458 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.458 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.458 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.458 DEBUG:boto:Data: 2024-03-04T16:37:24.458 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.458 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.458 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.458 DEBUG:boto:Params: {} 2024-03-04T16:37:24.458 DEBUG:boto:Token: None 2024-03-04T16:37:24.459 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.459 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008747c2ec537f7285-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.461 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.461 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.461 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.461 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.461 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.461 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.461 DEBUG:boto:Data: 2024-03-04T16:37:24.461 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.461 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.461 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.461 DEBUG:boto:Params: {} 2024-03-04T16:37:24.462 DEBUG:boto:Token: None 2024-03-04T16:37:24.462 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ecc01b0bbd6278cd-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.464 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.464 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.464 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.464 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.464 DEBUG:boto:Method: GET 2024-03-04T16:37:24.464 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.464 DEBUG:boto:Data: 2024-03-04T16:37:24.464 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.465 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.465 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.465 DEBUG:boto:Params: {} 2024-03-04T16:37:24.465 DEBUG:boto:Token: None 2024-03-04T16:37:24.465 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.465 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e609271e2dad66c3-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.467 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.467 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.467 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.467 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.467 DEBUG:boto:Method: GET 2024-03-04T16:37:24.467 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.468 DEBUG:boto:Data: 2024-03-04T16:37:24.468 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.468 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.468 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.468 DEBUG:boto:Params: {} 2024-03-04T16:37:24.468 DEBUG:boto:Token: None 2024-03-04T16:37:24.468 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b50326ef6fd89697-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.470 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.470 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.470 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.471 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.471 DEBUG:boto:Method: GET 2024-03-04T16:37:24.471 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.471 DEBUG:boto:Data: 2024-03-04T16:37:24.471 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.471 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.471 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.471 DEBUG:boto:Params: {} 2024-03-04T16:37:24.471 DEBUG:boto:Token: None 2024-03-04T16:37:24.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.471 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c00708d31bb0ae7-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.474 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.474 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.474 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.474 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.474 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.474 DEBUG:boto:Method: GET 2024-03-04T16:37:24.475 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.475 DEBUG:boto:Data: 2024-03-04T16:37:24.475 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.475 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.475 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.475 DEBUG:boto:Params: {} 2024-03-04T16:37:24.475 DEBUG:boto:Token: None 2024-03-04T16:37:24.475 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.475 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e120d43b601b431c-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.478 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.478 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.478 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.478 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.478 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.478 DEBUG:boto:Data: 2024-03-04T16:37:24.478 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.478 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.478 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.478 DEBUG:boto:Params: {} 2024-03-04T16:37:24.479 DEBUG:boto:Token: None 2024-03-04T16:37:24.479 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.479 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.480 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003896e3e9123ed9b1-0065e5f8c4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.481 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.481 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.481 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.481 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.481 DEBUG:boto:Data: 2024-03-04T16:37:24.481 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.481 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.481 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.481 DEBUG:boto:Params: {} 2024-03-04T16:37:24.481 DEBUG:boto:Token: None 2024-03-04T16:37:24.481 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.483 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005332e70c18f34967-0065e5f8c4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-25 zones={test-zone2, test-zone1} 2024-03-04T16:37:24.484 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-25 zones={test-zone3, test-zone1} 2024-03-04T16:37:24.484 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.484 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.484 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.484 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.484 DEBUG:boto:Data: 2024-03-04T16:37:24.484 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.484 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.484 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.484 DEBUG:boto:Params: {} 2024-03-04T16:37:24.485 DEBUG:boto:Token: None 2024-03-04T16:37:24.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.485 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f03b195caf1372c9-0065e5f8c4-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.487 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.487 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.487 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.487 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.487 DEBUG:boto:Data: 2024-03-04T16:37:24.487 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.488 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.488 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.488 DEBUG:boto:Params: {} 2024-03-04T16:37:24.488 DEBUG:boto:Token: None 2024-03-04T16:37:24.488 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.488 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000427fbb3795bacf59-0065e5f8c4-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:24.490 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.491 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.491 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.491 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.491 DEBUG:boto:Method: GET 2024-03-04T16:37:24.491 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.491 DEBUG:boto:Data: 2024-03-04T16:37:24.491 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.491 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.491 DEBUG:boto:Params: {} 2024-03-04T16:37:24.492 DEBUG:boto:Token: None 2024-03-04T16:37:24.492 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.492 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004baf6bbfa4d97815-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.494 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:24.495 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.495 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.495 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.495 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.495 DEBUG:boto:Method: GET 2024-03-04T16:37:24.495 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.496 DEBUG:boto:Data: 2024-03-04T16:37:24.496 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.496 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.496 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.496 DEBUG:boto:Params: {} 2024-03-04T16:37:24.496 DEBUG:boto:Token: None 2024-03-04T16:37:24.496 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.496 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a500eed32c84634-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.499 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.499 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.499 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.499 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.499 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.500 DEBUG:boto:Method: GET 2024-03-04T16:37:24.500 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.500 DEBUG:boto:Data: 2024-03-04T16:37:24.500 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.500 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.500 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.500 DEBUG:boto:Params: {} 2024-03-04T16:37:24.500 DEBUG:boto:Token: None 2024-03-04T16:37:24.500 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.500 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f9922c0ff430b0-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.503 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.503 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.503 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.503 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.503 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.503 DEBUG:boto:Method: GET 2024-03-04T16:37:24.503 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.504 DEBUG:boto:Data: 2024-03-04T16:37:24.504 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.504 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.504 DEBUG:boto:Params: {} 2024-03-04T16:37:24.504 DEBUG:boto:Token: None 2024-03-04T16:37:24.504 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.504 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67d0a13bc4fee28-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.507 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.507 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.507 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.507 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.507 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.507 DEBUG:boto:Data: 2024-03-04T16:37:24.507 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.508 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.508 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.508 DEBUG:boto:Params: {} 2024-03-04T16:37:24.508 DEBUG:boto:Token: None 2024-03-04T16:37:24.508 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.510 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b1fd47178e9a570d-0065e5f8c4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.510 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.510 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.510 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.510 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.510 DEBUG:boto:Data: 2024-03-04T16:37:24.510 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.511 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.511 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.511 DEBUG:boto:Params: {} 2024-03-04T16:37:24.511 DEBUG:boto:Token: None 2024-03-04T16:37:24.511 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.513 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004fd0846946099b76-0065e5f8c4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.513 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.513 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.513 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.514 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.514 DEBUG:boto:Method: GET 2024-03-04T16:37:24.514 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.514 DEBUG:boto:Data: 2024-03-04T16:37:24.514 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.514 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.514 DEBUG:boto:Params: {} 2024-03-04T16:37:24.514 DEBUG:boto:Token: None 2024-03-04T16:37:24.514 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a15c4cd0e001640-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.516 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.516 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.517 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.517 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.517 DEBUG:boto:Method: GET 2024-03-04T16:37:24.517 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.517 DEBUG:boto:Data: 2024-03-04T16:37:24.517 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.517 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.517 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.517 DEBUG:boto:Params: {} 2024-03-04T16:37:24.517 DEBUG:boto:Token: None 2024-03-04T16:37:24.518 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.518 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000959baf26e9a3fb46-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.520 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.520 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.520 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.520 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.520 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.520 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.520 DEBUG:boto:Data: 2024-03-04T16:37:24.520 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.520 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.521 DEBUG:boto:Params: {} 2024-03-04T16:37:24.521 DEBUG:boto:Token: None 2024-03-04T16:37:24.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000987291fd31d553b1-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.523 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.523 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.523 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.523 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.523 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.523 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.523 DEBUG:boto:Data: 2024-03-04T16:37:24.523 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.523 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.523 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.523 DEBUG:boto:Params: {} 2024-03-04T16:37:24.524 DEBUG:boto:Token: None 2024-03-04T16:37:24.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000632b242f3dfe308c-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.526 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.526 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.526 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.526 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.526 DEBUG:boto:Method: GET 2024-03-04T16:37:24.526 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.526 DEBUG:boto:Data: 2024-03-04T16:37:24.526 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.526 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.527 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.527 DEBUG:boto:Params: {} 2024-03-04T16:37:24.527 DEBUG:boto:Token: None 2024-03-04T16:37:24.527 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ecc425d6d0375f4-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.529 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.529 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.529 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.529 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.529 DEBUG:boto:Method: GET 2024-03-04T16:37:24.529 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.529 DEBUG:boto:Data: 2024-03-04T16:37:24.530 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.530 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.530 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.530 DEBUG:boto:Params: {} 2024-03-04T16:37:24.530 DEBUG:boto:Token: None 2024-03-04T16:37:24.530 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.530 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cc5427da20dccb6-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.532 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.532 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.532 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.532 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.532 DEBUG:boto:Method: GET 2024-03-04T16:37:24.532 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.533 DEBUG:boto:Data: 2024-03-04T16:37:24.533 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.533 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.533 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.533 DEBUG:boto:Params: {} 2024-03-04T16:37:24.533 DEBUG:boto:Token: None 2024-03-04T16:37:24.533 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0eacbd6ea3105df-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.576 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.576 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.576 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.576 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.577 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.577 DEBUG:boto:Method: GET 2024-03-04T16:37:24.577 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.577 DEBUG:boto:Data: 2024-03-04T16:37:24.577 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.577 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.577 DEBUG:boto:Params: {} 2024-03-04T16:37:24.577 DEBUG:boto:Token: None 2024-03-04T16:37:24.577 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004403ff4eb4ce448d-0065e5f8c4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.580 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.580 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.580 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.580 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.581 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.581 DEBUG:boto:Data: 2024-03-04T16:37:24.581 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.581 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.581 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.581 DEBUG:boto:Params: {} 2024-03-04T16:37:24.581 DEBUG:boto:Token: None 2024-03-04T16:37:24.581 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.581 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.583 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007e51dbb367182633-0065e5f8c4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.583 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.583 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.583 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.584 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.584 DEBUG:boto:Data: 2024-03-04T16:37:24.584 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.584 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:24.584 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.584 DEBUG:boto:Params: {} 2024-03-04T16:37:24.584 DEBUG:boto:Token: None 2024-03-04T16:37:24.584 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.584 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.586 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000003dec66c1deee4d6-0065e5f8c4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-25 zones={test-zone3, test-zone1} 2024-03-04T16:37:24.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-25 zones={test-zone3, test-zone2} 2024-03-04T16:37:24.587 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.587 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.587 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.587 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.587 DEBUG:boto:Data: 2024-03-04T16:37:24.587 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.587 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.587 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.587 DEBUG:boto:Params: {} 2024-03-04T16:37:24.587 DEBUG:boto:Token: None 2024-03-04T16:37:24.587 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.588 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb32b2ccee2a59a5-0065e5f8c4-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.590 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.590 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.590 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.590 DEBUG:boto:Path: /sbnvju-25/ 2024-03-04T16:37:24.590 DEBUG:boto:Data: 2024-03-04T16:37:24.590 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.590 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.590 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.590 DEBUG:boto:Params: {} 2024-03-04T16:37:24.591 DEBUG:boto:Token: None 2024-03-04T16:37:24.591 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.591 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4= 2024-03-04T16:37:24.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uPvqqOZu9ErX9cv/2lrCtEzhcx4=', 'Content-Length': '0'} 2024-03-04T16:37:24.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f532bb7fd6a080a5-0065e5f8c4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:24.593 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.593 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.593 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.593 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.593 DEBUG:boto:Method: GET 2024-03-04T16:37:24.593 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.594 DEBUG:boto:Data: 2024-03-04T16:37:24.594 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.594 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.594 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.594 DEBUG:boto:Params: {} 2024-03-04T16:37:24.594 DEBUG:boto:Token: None 2024-03-04T16:37:24.594 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006877e6730d9729a4-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.597 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:24.597 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.597 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.597 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.598 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.598 DEBUG:boto:Method: GET 2024-03-04T16:37:24.598 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.598 DEBUG:boto:Data: 2024-03-04T16:37:24.598 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.598 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.598 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.598 DEBUG:boto:Params: {} 2024-03-04T16:37:24.598 DEBUG:boto:Token: None 2024-03-04T16:37:24.598 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.598 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007295436c80a7c0a3-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.644 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:24.645 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.645 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.645 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.645 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.645 DEBUG:boto:Method: GET 2024-03-04T16:37:24.645 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.646 DEBUG:boto:Data: 2024-03-04T16:37:24.646 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.646 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.646 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.646 DEBUG:boto:Params: {} 2024-03-04T16:37:24.646 DEBUG:boto:Token: None 2024-03-04T16:37:24.646 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.646 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd158fb850ca57af-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.649 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.649 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.649 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.649 DEBUG:boto:path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.650 DEBUG:boto:auth_path=/sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.650 DEBUG:boto:Method: GET 2024-03-04T16:37:24.650 DEBUG:boto:Path: /sbnvju-25/?versions&max-keys=999 2024-03-04T16:37:24.650 DEBUG:boto:Data: 2024-03-04T16:37:24.650 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.650 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.650 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.650 DEBUG:boto:Params: {} 2024-03-04T16:37:24.650 DEBUG:boto:Token: None 2024-03-04T16:37:24.650 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/?versions 2024-03-04T16:37:24.650 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8= 2024-03-04T16:37:24.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pvcG9rqyVtQUWmy5Sf34PHOqDt8=', 'Content-Length': '0'} 2024-03-04T16:37:24.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b38a9477d0134cc-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.653 DEBUG:boto:b'sbnvju-25999falseobjfpwbOan4pTZ3sglQ4vZBwDaFvwwq-RFtrue2024-03-04T16:36:37.573Zrgw-multisite-test-userMultisite Test UserNormalobj2TqpINDbBnhqAGsptIELrwOcTKXGrptfalse2024-03-04T16:36:37.524Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:24.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.653 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.653 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.653 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.653 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.654 DEBUG:boto:Data: 2024-03-04T16:37:24.654 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.654 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.654 DEBUG:boto:Params: {} 2024-03-04T16:37:24.654 DEBUG:boto:Token: None 2024-03-04T16:37:24.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.654 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.656 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000012e237103791b14a-0065e5f8c4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.656 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.656 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.656 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.656 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.656 DEBUG:boto:Data: 2024-03-04T16:37:24.656 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.656 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.657 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.657 DEBUG:boto:Params: {} 2024-03-04T16:37:24.657 DEBUG:boto:Token: None 2024-03-04T16:37:24.657 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.657 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.659 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d609f61977d39be8-0065e5f8c4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.659 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.659 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.659 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.659 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.660 DEBUG:boto:Method: GET 2024-03-04T16:37:24.660 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.660 DEBUG:boto:Data: 2024-03-04T16:37:24.660 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.660 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.660 DEBUG:boto:Params: {} 2024-03-04T16:37:24.660 DEBUG:boto:Token: None 2024-03-04T16:37:24.660 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000878274586d41419f-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.663 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.663 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.663 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.663 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.663 DEBUG:boto:Method: GET 2024-03-04T16:37:24.663 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.663 DEBUG:boto:Data: 2024-03-04T16:37:24.663 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.663 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.663 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.663 DEBUG:boto:Params: {} 2024-03-04T16:37:24.663 DEBUG:boto:Token: None 2024-03-04T16:37:24.664 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.664 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f71e3105e1d762c-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.665 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.666 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.666 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.666 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.666 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.666 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.666 DEBUG:boto:Data: 2024-03-04T16:37:24.666 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.666 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.666 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.666 DEBUG:boto:Params: {} 2024-03-04T16:37:24.666 DEBUG:boto:Token: None 2024-03-04T16:37:24.666 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.667 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000437db05d5395ddb4-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.668 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.669 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.669 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.669 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.669 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.669 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.669 DEBUG:boto:Data: 2024-03-04T16:37:24.669 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.669 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.669 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.669 DEBUG:boto:Params: {} 2024-03-04T16:37:24.669 DEBUG:boto:Token: None 2024-03-04T16:37:24.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.670 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c= 2024-03-04T16:37:24.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RVjiAP8fauPB8Tdtoh8Y57roY5c=', 'Content-Length': '0'} 2024-03-04T16:37:24.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b77bc5eb6037b79-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:24.672 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.672 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.672 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.672 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.672 DEBUG:boto:Method: GET 2024-03-04T16:37:24.672 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.672 DEBUG:boto:Data: 2024-03-04T16:37:24.672 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.672 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.673 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.673 DEBUG:boto:Params: {} 2024-03-04T16:37:24.673 DEBUG:boto:Token: None 2024-03-04T16:37:24.673 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.673 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a1482fc7-8de0-4f7d-8e05-b02ecec7d664:sbnvju-25:fc294220-5038-4f25-aeee-cac2c5838729.4341.34'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026f6e1d318eceb41-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.675 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.675 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.675 DEBUG:boto:path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.675 DEBUG:boto:auth_path=/sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.675 DEBUG:boto:Method: GET 2024-03-04T16:37:24.675 DEBUG:boto:Path: /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.675 DEBUG:boto:Data: 2024-03-04T16:37:24.676 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.676 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.676 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.676 DEBUG:boto:Params: {} 2024-03-04T16:37:24.676 DEBUG:boto:Token: None 2024-03-04T16:37:24.676 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj?versionId=2TqpINDbBnhqAGsptIELrwOcTKXGrpt 2024-03-04T16:37:24.676 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ= 2024-03-04T16:37:24.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xFrY6w7KCBKJ5plfnGN0kL9lJlQ=', 'Content-Length': '0'} 2024-03-04T16:37:24.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', '2TqpINDbBnhqAGsptIELrwOcTKXGrpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000045e500c06398d99-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.678 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.678 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.679 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.679 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.679 DEBUG:boto:Method: GET 2024-03-04T16:37:24.679 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.679 DEBUG:boto:Data: 2024-03-04T16:37:24.679 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.679 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.679 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.679 DEBUG:boto:Params: {} 2024-03-04T16:37:24.679 DEBUG:boto:Token: None 2024-03-04T16:37:24.679 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.680 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037972e6aee74e83f-0065e5f8c4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.724 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.724 DEBUG:boto:path=/sbnvju-25/ 2024-03-04T16:37:24.724 DEBUG:boto:auth_path=/sbnvju-25/ 2024-03-04T16:37:24.725 DEBUG:boto:path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.725 DEBUG:boto:auth_path=/sbnvju-25/?prefix=obj 2024-03-04T16:37:24.725 DEBUG:boto:Method: GET 2024-03-04T16:37:24.725 DEBUG:boto:Path: /sbnvju-25/?prefix=obj 2024-03-04T16:37:24.725 DEBUG:boto:Data: 2024-03-04T16:37:24.725 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.725 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.725 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.725 DEBUG:boto:Params: {} 2024-03-04T16:37:24.725 DEBUG:boto:Token: None 2024-03-04T16:37:24.726 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/ 2024-03-04T16:37:24.726 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8= 2024-03-04T16:37:24.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+ObcQ3G7NGHRKT25VowGs/ZF9E8=', 'Content-Length': '0'} 2024-03-04T16:37:24.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5ed2eb4d58e1f8-0065e5f8c4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.728 DEBUG:boto:b'sbnvju-25obj1000false' 2024-03-04T16:37:24.728 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.728 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.728 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.729 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.729 DEBUG:boto:Data: 2024-03-04T16:37:24.729 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.729 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:24.729 DEBUG:boto:Port: 8001 2024-03-04T16:37:24.729 DEBUG:boto:Params: {} 2024-03-04T16:37:24.729 DEBUG:boto:Token: None 2024-03-04T16:37:24.729 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.729 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.731 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008c95aa54396e9406-0065e5f8c4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.731 DEBUG:boto:path=/sbnvju-25/obj 2024-03-04T16:37:24.731 DEBUG:boto:auth_path=/sbnvju-25/obj 2024-03-04T16:37:24.732 DEBUG:boto:Method: HEAD 2024-03-04T16:37:24.732 DEBUG:boto:Path: /sbnvju-25/obj 2024-03-04T16:37:24.732 DEBUG:boto:Data: 2024-03-04T16:37:24.732 DEBUG:boto:Headers: {} 2024-03-04T16:37:24.732 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:24.732 DEBUG:boto:Port: 8000 2024-03-04T16:37:24.732 DEBUG:boto:Params: {} 2024-03-04T16:37:24.732 DEBUG:boto:Token: None 2024-03-04T16:37:24.732 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:24 GMT /sbnvju-25/obj 2024-03-04T16:37:24.732 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90= 2024-03-04T16:37:24.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:24 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NRuHyZ03dxcpYBZe+o0pgg83Q90=', 'Content-Length': '0'} 2024-03-04T16:37:24.734 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007785ede09572673c-0065e5f8c4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:24 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:24.734 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-25 zones={test-zone3, test-zone2} 2024-03-04T16:37:24.734 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-26 2024-03-04T16:37:24.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.735 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:24.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:24.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:24.851+0000 7f47fde67800 1 bucket sync caught up with source: 2024-03-04T16:37:24.839 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000002.57.6, , ] 2024-03-04T16:37:24.839 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000002.57.6, , ] 2024-03-04T16:37:24.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:24.851+0000 7f47fde67800 0 bucket checkpoint complete 2024-03-04T16:37:24.856 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-26 2024-03-04T16:37:24.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.857 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:24.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.978+0000 7f7408511800 1 bucket sync caught up with source: 2024-03-04T16:37:24.966 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000002.57.6, , ] 2024-03-04T16:37:24.966 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000002.57.6, , ] 2024-03-04T16:37:24.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:24.978+0000 7f7408511800 0 bucket checkpoint complete 2024-03-04T16:37:24.985 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-26 2024-03-04T16:37:24.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:24.986 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:25.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:25.104+0000 7fb99d330800 1 waiting to reach incremental sync.. 2024-03-04T16:37:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:30.105+0000 7fb99d330800 1 waiting to reach incremental sync.. 2024-03-04T16:37:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:35.106+0000 7fb99d330800 1 waiting to reach incremental sync.. 2024-03-04T16:37:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.108+0000 7fb99d330800 1 bucket sync caught up with source: 2024-03-04T16:37:40.096 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000002.61.6, , ] 2024-03-04T16:37:40.096 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000002.61.6, , ] 2024-03-04T16:37:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.108+0000 7fb99d330800 0 bucket checkpoint complete 2024-03-04T16:37:40.120 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-26 2024-03-04T16:37:40.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.120 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:40.178 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:40.178 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.245+0000 7f1def724800 1 bucket sync caught up with source: 2024-03-04T16:37:40.232 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000002.61.6, , ] 2024-03-04T16:37:40.232 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000002.61.6, , ] 2024-03-04T16:37:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.245+0000 7f1def724800 0 bucket checkpoint complete 2024-03-04T16:37:40.252 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-26 2024-03-04T16:37:40.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.252 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:40.306 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:40.306 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.369+0000 7f120a34b800 1 bucket sync caught up with source: 2024-03-04T16:37:40.356 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.101.12, , ] 2024-03-04T16:37:40.356 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.101.12, , ] 2024-03-04T16:37:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:40.369+0000 7f120a34b800 0 bucket checkpoint complete 2024-03-04T16:37:40.376 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-26 2024-03-04T16:37:40.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:40.376 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:40.437 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:40.437 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:40.489 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:40.501+0000 7f1eb7f2f800 1 bucket sync caught up with source: 2024-03-04T16:37:40.489 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.101.12, , ] 2024-03-04T16:37:40.489 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.101.12, , ] 2024-03-04T16:37:40.489 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:40.501+0000 7f1eb7f2f800 0 bucket checkpoint complete 2024-03-04T16:37:40.508 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-26 zones={test-zone2, test-zone1} 2024-03-04T16:37:40.508 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.508 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.508 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.508 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.508 DEBUG:boto:Data: 2024-03-04T16:37:40.508 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.508 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.509 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.509 DEBUG:boto:Params: {} 2024-03-04T16:37:40.509 DEBUG:boto:Token: None 2024-03-04T16:37:40.509 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ce50b42aef8fa11-0065e5f8d4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.512 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.512 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.512 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.512 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.512 DEBUG:boto:Data: 2024-03-04T16:37:40.512 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.512 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.512 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.512 DEBUG:boto:Params: {} 2024-03-04T16:37:40.513 DEBUG:boto:Token: None 2024-03-04T16:37:40.513 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067a0878b19c501b7-0065e5f8d4-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.515 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:40.515 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.515 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.515 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.515 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.516 DEBUG:boto:Method: GET 2024-03-04T16:37:40.516 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.516 DEBUG:boto:Data: 2024-03-04T16:37:40.516 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.516 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.516 DEBUG:boto:Params: {} 2024-03-04T16:37:40.516 DEBUG:boto:Token: None 2024-03-04T16:37:40.516 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.516 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773ca7030100926a-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.519 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.519 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:40.520 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.520 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.520 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.520 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.520 DEBUG:boto:Method: GET 2024-03-04T16:37:40.520 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.520 DEBUG:boto:Data: 2024-03-04T16:37:40.520 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.520 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.520 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.520 DEBUG:boto:Params: {} 2024-03-04T16:37:40.521 DEBUG:boto:Token: None 2024-03-04T16:37:40.521 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5534050040fcb9-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.523 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.524 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.524 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.524 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.524 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.524 DEBUG:boto:Method: GET 2024-03-04T16:37:40.524 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.524 DEBUG:boto:Data: 2024-03-04T16:37:40.525 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.525 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.525 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.525 DEBUG:boto:Params: {} 2024-03-04T16:37:40.525 DEBUG:boto:Token: None 2024-03-04T16:37:40.525 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f703ceb259f977a-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.528 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.528 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.528 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.528 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.528 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.528 DEBUG:boto:Method: GET 2024-03-04T16:37:40.528 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.528 DEBUG:boto:Data: 2024-03-04T16:37:40.529 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.529 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.529 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.529 DEBUG:boto:Params: {} 2024-03-04T16:37:40.529 DEBUG:boto:Token: None 2024-03-04T16:37:40.529 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.529 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029e27d7481ab8f6f-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.531 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.532 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.532 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.532 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.532 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.532 DEBUG:boto:Data: 2024-03-04T16:37:40.532 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.532 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.532 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.532 DEBUG:boto:Params: {} 2024-03-04T16:37:40.533 DEBUG:boto:Token: None 2024-03-04T16:37:40.533 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.533 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.535 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000447ec062a91750d3-0065e5f8d4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.535 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.535 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.535 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.535 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.535 DEBUG:boto:Data: 2024-03-04T16:37:40.535 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.535 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.535 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.535 DEBUG:boto:Params: {} 2024-03-04T16:37:40.535 DEBUG:boto:Token: None 2024-03-04T16:37:40.536 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.536 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.538 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000089c2bfe37a7f3b1c-0065e5f8d4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.538 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.538 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.538 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.538 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.538 DEBUG:boto:Method: GET 2024-03-04T16:37:40.538 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.538 DEBUG:boto:Data: 2024-03-04T16:37:40.538 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.539 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.539 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.539 DEBUG:boto:Params: {} 2024-03-04T16:37:40.539 DEBUG:boto:Token: None 2024-03-04T16:37:40.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002dbd423211956aaa-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.541 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.541 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.541 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.541 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.541 DEBUG:boto:Method: GET 2024-03-04T16:37:40.541 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.541 DEBUG:boto:Data: 2024-03-04T16:37:40.541 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.542 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.542 DEBUG:boto:Params: {} 2024-03-04T16:37:40.542 DEBUG:boto:Token: None 2024-03-04T16:37:40.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034b8becac709c10e-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.544 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.545 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.545 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.545 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.545 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.545 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.545 DEBUG:boto:Data: 2024-03-04T16:37:40.545 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.545 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.545 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.545 DEBUG:boto:Params: {} 2024-03-04T16:37:40.545 DEBUG:boto:Token: None 2024-03-04T16:37:40.545 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c529dc6ae9e79102-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.547 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.547 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.548 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.548 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.548 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.548 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.548 DEBUG:boto:Data: 2024-03-04T16:37:40.548 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.548 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.548 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.548 DEBUG:boto:Params: {} 2024-03-04T16:37:40.548 DEBUG:boto:Token: None 2024-03-04T16:37:40.548 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000164beb974b80f8d-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.550 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.551 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.551 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.551 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.551 DEBUG:boto:Method: GET 2024-03-04T16:37:40.551 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.551 DEBUG:boto:Data: 2024-03-04T16:37:40.551 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.551 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.551 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.551 DEBUG:boto:Params: {} 2024-03-04T16:37:40.551 DEBUG:boto:Token: None 2024-03-04T16:37:40.551 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008683c8e1d1e5cb33-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.554 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.554 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.554 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.554 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.554 DEBUG:boto:Method: GET 2024-03-04T16:37:40.554 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.554 DEBUG:boto:Data: 2024-03-04T16:37:40.554 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.554 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.555 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.555 DEBUG:boto:Params: {} 2024-03-04T16:37:40.555 DEBUG:boto:Token: None 2024-03-04T16:37:40.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.555 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040287fe3c527aaaa-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.557 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.557 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.557 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.557 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.558 DEBUG:boto:Method: GET 2024-03-04T16:37:40.558 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.558 DEBUG:boto:Data: 2024-03-04T16:37:40.558 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.558 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.558 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.558 DEBUG:boto:Params: {} 2024-03-04T16:37:40.558 DEBUG:boto:Token: None 2024-03-04T16:37:40.558 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.558 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50949d0b578625d-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.561 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.561 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.561 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.561 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.561 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.561 DEBUG:boto:Method: GET 2024-03-04T16:37:40.561 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.562 DEBUG:boto:Data: 2024-03-04T16:37:40.562 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.562 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.562 DEBUG:boto:Params: {} 2024-03-04T16:37:40.562 DEBUG:boto:Token: None 2024-03-04T16:37:40.562 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b4946556243873-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.564 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.565 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.565 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.565 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.565 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.565 DEBUG:boto:Data: 2024-03-04T16:37:40.565 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.565 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.565 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.565 DEBUG:boto:Params: {} 2024-03-04T16:37:40.565 DEBUG:boto:Token: None 2024-03-04T16:37:40.566 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.568 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000dbd32f6e1c54b36a-0065e5f8d4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.568 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.568 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.568 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.568 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.568 DEBUG:boto:Data: 2024-03-04T16:37:40.568 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.568 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.568 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.569 DEBUG:boto:Params: {} 2024-03-04T16:37:40.569 DEBUG:boto:Token: None 2024-03-04T16:37:40.569 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.569 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.571 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005ae9ea13fa6749a1-0065e5f8d4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-26 zones={test-zone2, test-zone1} 2024-03-04T16:37:40.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-26 zones={test-zone3, test-zone1} 2024-03-04T16:37:40.571 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.571 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.571 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.571 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.571 DEBUG:boto:Data: 2024-03-04T16:37:40.571 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.571 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.571 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.572 DEBUG:boto:Params: {} 2024-03-04T16:37:40.572 DEBUG:boto:Token: None 2024-03-04T16:37:40.572 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.572 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002261af916d149979-0065e5f8d4-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.574 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.574 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.574 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.575 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.575 DEBUG:boto:Data: 2024-03-04T16:37:40.575 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.575 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.575 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.575 DEBUG:boto:Params: {} 2024-03-04T16:37:40.575 DEBUG:boto:Token: None 2024-03-04T16:37:40.575 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3224dc3d5b88b97-0065e5f8d4-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:40.577 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.577 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.578 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.578 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.578 DEBUG:boto:Method: GET 2024-03-04T16:37:40.578 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.578 DEBUG:boto:Data: 2024-03-04T16:37:40.578 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.578 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.578 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.578 DEBUG:boto:Params: {} 2024-03-04T16:37:40.578 DEBUG:boto:Token: None 2024-03-04T16:37:40.578 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.579 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07c1ee567b3d597-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.581 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:40.581 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.582 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.582 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.582 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.582 DEBUG:boto:Method: GET 2024-03-04T16:37:40.582 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.582 DEBUG:boto:Data: 2024-03-04T16:37:40.582 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.582 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.582 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.582 DEBUG:boto:Params: {} 2024-03-04T16:37:40.582 DEBUG:boto:Token: None 2024-03-04T16:37:40.583 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.583 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9cae0b465a40d07-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.628 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.629 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.629 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.629 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.629 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.629 DEBUG:boto:Method: GET 2024-03-04T16:37:40.629 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.629 DEBUG:boto:Data: 2024-03-04T16:37:40.629 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.629 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.629 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.629 DEBUG:boto:Params: {} 2024-03-04T16:37:40.630 DEBUG:boto:Token: None 2024-03-04T16:37:40.630 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.630 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b6e139ca1ed018-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.632 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.632 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.633 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.633 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.633 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.633 DEBUG:boto:Method: GET 2024-03-04T16:37:40.633 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.633 DEBUG:boto:Data: 2024-03-04T16:37:40.633 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.633 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.633 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.633 DEBUG:boto:Params: {} 2024-03-04T16:37:40.633 DEBUG:boto:Token: None 2024-03-04T16:37:40.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7dad730219d057-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.636 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.637 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.637 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.637 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.637 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.637 DEBUG:boto:Data: 2024-03-04T16:37:40.637 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.637 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.637 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.637 DEBUG:boto:Params: {} 2024-03-04T16:37:40.637 DEBUG:boto:Token: None 2024-03-04T16:37:40.638 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.639 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cde3d5f89a5d6702-0065e5f8d4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.639 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.640 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.640 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.640 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.640 DEBUG:boto:Data: 2024-03-04T16:37:40.640 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.640 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.640 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.640 DEBUG:boto:Params: {} 2024-03-04T16:37:40.640 DEBUG:boto:Token: None 2024-03-04T16:37:40.641 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.641 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.642 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b5b3cbd2874869ea-0065e5f8d4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.643 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.643 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.643 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.643 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.643 DEBUG:boto:Method: GET 2024-03-04T16:37:40.643 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.644 DEBUG:boto:Data: 2024-03-04T16:37:40.644 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.644 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.644 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.644 DEBUG:boto:Params: {} 2024-03-04T16:37:40.645 DEBUG:boto:Token: None 2024-03-04T16:37:40.645 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.645 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000252f7f9c0e086088-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.647 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.647 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.647 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.647 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.647 DEBUG:boto:Method: GET 2024-03-04T16:37:40.648 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.648 DEBUG:boto:Data: 2024-03-04T16:37:40.648 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.648 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.648 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.648 DEBUG:boto:Params: {} 2024-03-04T16:37:40.648 DEBUG:boto:Token: None 2024-03-04T16:37:40.648 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.648 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f608eaf9aeec7b7-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.650 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.650 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.650 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.651 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.651 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.651 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.651 DEBUG:boto:Data: 2024-03-04T16:37:40.651 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.651 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.651 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.651 DEBUG:boto:Params: {} 2024-03-04T16:37:40.651 DEBUG:boto:Token: None 2024-03-04T16:37:40.651 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.651 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcc1d2c1bee4077a-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.653 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.653 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.653 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.654 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.654 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.654 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.654 DEBUG:boto:Data: 2024-03-04T16:37:40.654 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.654 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.654 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.654 DEBUG:boto:Params: {} 2024-03-04T16:37:40.654 DEBUG:boto:Token: None 2024-03-04T16:37:40.654 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.655 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008172156ecbf40dc2-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.656 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.656 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.657 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.657 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.657 DEBUG:boto:Method: GET 2024-03-04T16:37:40.657 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.657 DEBUG:boto:Data: 2024-03-04T16:37:40.657 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.657 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.657 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.657 DEBUG:boto:Params: {} 2024-03-04T16:37:40.657 DEBUG:boto:Token: None 2024-03-04T16:37:40.657 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.658 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003cdb47514ce213c7-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.659 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.659 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.659 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.659 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.659 DEBUG:boto:Method: GET 2024-03-04T16:37:40.660 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.660 DEBUG:boto:Data: 2024-03-04T16:37:40.660 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.660 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.660 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.660 DEBUG:boto:Params: {} 2024-03-04T16:37:40.660 DEBUG:boto:Token: None 2024-03-04T16:37:40.660 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.660 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dff2e6640834424-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.662 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.662 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.662 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.662 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.662 DEBUG:boto:Method: GET 2024-03-04T16:37:40.662 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.663 DEBUG:boto:Data: 2024-03-04T16:37:40.663 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.663 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.663 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.663 DEBUG:boto:Params: {} 2024-03-04T16:37:40.663 DEBUG:boto:Token: None 2024-03-04T16:37:40.663 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.663 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c469806cc35f2dd-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.666 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.666 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.666 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.666 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.666 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.666 DEBUG:boto:Method: GET 2024-03-04T16:37:40.666 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.666 DEBUG:boto:Data: 2024-03-04T16:37:40.666 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.667 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.667 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.667 DEBUG:boto:Params: {} 2024-03-04T16:37:40.667 DEBUG:boto:Token: None 2024-03-04T16:37:40.667 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.667 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966b7507f7f55225-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.712 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.712 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.712 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.713 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.713 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.713 DEBUG:boto:Data: 2024-03-04T16:37:40.713 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.713 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.713 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.713 DEBUG:boto:Params: {} 2024-03-04T16:37:40.713 DEBUG:boto:Token: None 2024-03-04T16:37:40.713 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.713 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.715 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007dfa801899f79bb1-0065e5f8d4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.716 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.716 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.716 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.716 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.716 DEBUG:boto:Data: 2024-03-04T16:37:40.716 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.716 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.716 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.716 DEBUG:boto:Params: {} 2024-03-04T16:37:40.717 DEBUG:boto:Token: None 2024-03-04T16:37:40.717 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.717 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.718 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000030d5205e2d2c2a8e-0065e5f8d4-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.719 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-26 zones={test-zone3, test-zone1} 2024-03-04T16:37:40.719 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-26 zones={test-zone3, test-zone2} 2024-03-04T16:37:40.719 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.719 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.719 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.719 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.719 DEBUG:boto:Data: 2024-03-04T16:37:40.719 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.719 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.719 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.720 DEBUG:boto:Params: {} 2024-03-04T16:37:40.720 DEBUG:boto:Token: None 2024-03-04T16:37:40.720 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.720 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077c6a68f46a95715-0065e5f8d4-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.722 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.722 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.722 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.722 DEBUG:boto:Path: /sbnvju-26/ 2024-03-04T16:37:40.722 DEBUG:boto:Data: 2024-03-04T16:37:40.723 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.723 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.723 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.723 DEBUG:boto:Params: {} 2024-03-04T16:37:40.723 DEBUG:boto:Token: None 2024-03-04T16:37:40.723 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.723 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0= 2024-03-04T16:37:40.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VWquD0xRlAHAkPR+Erz979WofZ0=', 'Content-Length': '0'} 2024-03-04T16:37:40.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac514ffceb725413-0065e5f8d4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:37:40.725 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.726 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.726 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.726 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.726 DEBUG:boto:Method: GET 2024-03-04T16:37:40.726 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.726 DEBUG:boto:Data: 2024-03-04T16:37:40.726 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.726 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.726 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.726 DEBUG:boto:Params: {} 2024-03-04T16:37:40.726 DEBUG:boto:Token: None 2024-03-04T16:37:40.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c733c92294e8284a-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.729 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:37:40.730 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.730 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.730 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.730 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.730 DEBUG:boto:Method: GET 2024-03-04T16:37:40.730 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.730 DEBUG:boto:Data: 2024-03-04T16:37:40.730 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.730 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.730 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.730 DEBUG:boto:Params: {} 2024-03-04T16:37:40.731 DEBUG:boto:Token: None 2024-03-04T16:37:40.731 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.731 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9857bfa4f40e36-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.776 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:37:40.777 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.777 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.777 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.777 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.777 DEBUG:boto:Method: GET 2024-03-04T16:37:40.777 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.777 DEBUG:boto:Data: 2024-03-04T16:37:40.777 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.777 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.777 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.778 DEBUG:boto:Params: {} 2024-03-04T16:37:40.778 DEBUG:boto:Token: None 2024-03-04T16:37:40.778 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000727d9d305fc0ff33-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.824 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.824 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.825 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.825 DEBUG:boto:path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.825 DEBUG:boto:auth_path=/sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.825 DEBUG:boto:Method: GET 2024-03-04T16:37:40.825 DEBUG:boto:Path: /sbnvju-26/?versions&max-keys=999 2024-03-04T16:37:40.825 DEBUG:boto:Data: 2024-03-04T16:37:40.825 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.825 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.825 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.825 DEBUG:boto:Params: {} 2024-03-04T16:37:40.826 DEBUG:boto:Token: None 2024-03-04T16:37:40.826 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/?versions 2024-03-04T16:37:40.826 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20= 2024-03-04T16:37:40.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ndUXPRSnV97ghCNpIXkP3Do8A20=', 'Content-Length': '0'} 2024-03-04T16:37:40.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008963d8ec4edfd0af-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.828 DEBUG:boto:b'sbnvju-26999falseobjkSy06mbdT5st3Ao6UR-UOYGVJSFhd2rtrue2024-03-04T16:36:37.601Zrgw-multisite-test-userMultisite Test UserNormalobjG-Cqf5vRejnn10ihd33QcbtWHS6e2nffalse2024-03-04T16:36:37.586Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:37:40.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.829 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.829 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.829 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.829 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.829 DEBUG:boto:Data: 2024-03-04T16:37:40.829 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.829 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.829 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.830 DEBUG:boto:Params: {} 2024-03-04T16:37:40.830 DEBUG:boto:Token: None 2024-03-04T16:37:40.830 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.830 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.832 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c2db51e10e4f2666-0065e5f8d4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.832 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.832 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.832 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.832 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.832 DEBUG:boto:Data: 2024-03-04T16:37:40.832 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.832 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.832 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.833 DEBUG:boto:Params: {} 2024-03-04T16:37:40.833 DEBUG:boto:Token: None 2024-03-04T16:37:40.833 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.833 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.835 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003e4cb4944294f3a0-0065e5f8d4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.835 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.835 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.836 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.836 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.836 DEBUG:boto:Method: GET 2024-03-04T16:37:40.836 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.836 DEBUG:boto:Data: 2024-03-04T16:37:40.836 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.836 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.836 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.836 DEBUG:boto:Params: {} 2024-03-04T16:37:40.836 DEBUG:boto:Token: None 2024-03-04T16:37:40.837 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.837 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4c8c6ba5fbd55fa-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.839 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.839 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.839 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.839 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.839 DEBUG:boto:Method: GET 2024-03-04T16:37:40.839 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.839 DEBUG:boto:Data: 2024-03-04T16:37:40.839 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.839 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.839 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.839 DEBUG:boto:Params: {} 2024-03-04T16:37:40.840 DEBUG:boto:Token: None 2024-03-04T16:37:40.840 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.840 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ee0f3dd69d22e08-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.842 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.842 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.842 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.842 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.842 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.842 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.842 DEBUG:boto:Data: 2024-03-04T16:37:40.843 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.843 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.843 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.843 DEBUG:boto:Params: {} 2024-03-04T16:37:40.843 DEBUG:boto:Token: None 2024-03-04T16:37:40.843 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.843 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003865d7d36ae25ac8-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.845 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.845 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.845 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.845 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.845 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.845 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.846 DEBUG:boto:Data: 2024-03-04T16:37:40.846 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.846 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.846 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.846 DEBUG:boto:Params: {} 2024-03-04T16:37:40.846 DEBUG:boto:Token: None 2024-03-04T16:37:40.846 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.846 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U= 2024-03-04T16:37:40.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KiOEFGW3D77ZuebKxo6xjRU6K9U=', 'Content-Length': '0'} 2024-03-04T16:37:40.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab6ca76b2716e2f4-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:37:40.848 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.848 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.848 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.848 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.849 DEBUG:boto:Method: GET 2024-03-04T16:37:40.849 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.849 DEBUG:boto:Data: 2024-03-04T16:37:40.849 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.849 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.849 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.849 DEBUG:boto:Params: {} 2024-03-04T16:37:40.849 DEBUG:boto:Token: None 2024-03-04T16:37:40.849 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.849 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005707ad2a95d62da6-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.851 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.851 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.852 DEBUG:boto:path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.852 DEBUG:boto:auth_path=/sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.852 DEBUG:boto:Method: GET 2024-03-04T16:37:40.852 DEBUG:boto:Path: /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.852 DEBUG:boto:Data: 2024-03-04T16:37:40.852 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.852 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.852 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.852 DEBUG:boto:Params: {} 2024-03-04T16:37:40.852 DEBUG:boto:Token: None 2024-03-04T16:37:40.853 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj?versionId=G-Cqf5vRejnn10ihd33QcbtWHS6e2nf 2024-03-04T16:37:40.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk= 2024-03-04T16:37:40.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VHYJYk5WJ5DNj5gQ+Bh0ZkfN6Tk=', 'Content-Length': '0'} 2024-03-04T16:37:40.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 04 Mar 2024 16:36:37 GMT'), ('x-amz-version-id', 'G-Cqf5vRejnn10ihd33QcbtWHS6e2nf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'dce1012f-1cf8-44d9-b80b-aa1da8d03699:sbnvju-26:fc294220-5038-4f25-aeee-cac2c5838729.4341.35'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b16fd777859b811-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.855 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.855 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.855 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.855 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.855 DEBUG:boto:Method: GET 2024-03-04T16:37:40.855 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.855 DEBUG:boto:Data: 2024-03-04T16:37:40.855 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.855 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.855 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.856 DEBUG:boto:Params: {} 2024-03-04T16:37:40.856 DEBUG:boto:Token: None 2024-03-04T16:37:40.856 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.856 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b29d521ca2811623-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.859 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.859 DEBUG:boto:path=/sbnvju-26/ 2024-03-04T16:37:40.859 DEBUG:boto:auth_path=/sbnvju-26/ 2024-03-04T16:37:40.859 DEBUG:boto:path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.859 DEBUG:boto:auth_path=/sbnvju-26/?prefix=obj 2024-03-04T16:37:40.859 DEBUG:boto:Method: GET 2024-03-04T16:37:40.859 DEBUG:boto:Path: /sbnvju-26/?prefix=obj 2024-03-04T16:37:40.859 DEBUG:boto:Data: 2024-03-04T16:37:40.859 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.860 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.860 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.860 DEBUG:boto:Params: {} 2024-03-04T16:37:40.860 DEBUG:boto:Token: None 2024-03-04T16:37:40.860 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/ 2024-03-04T16:37:40.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM= 2024-03-04T16:37:40.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CiZx2gOf2V8cuGDLd4giuD4UBCM=', 'Content-Length': '0'} 2024-03-04T16:37:40.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001baece52741c3964-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.862 DEBUG:boto:b'sbnvju-26obj1000false' 2024-03-04T16:37:40.862 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.863 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.863 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.863 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.863 DEBUG:boto:Data: 2024-03-04T16:37:40.863 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.863 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.863 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.863 DEBUG:boto:Params: {} 2024-03-04T16:37:40.863 DEBUG:boto:Token: None 2024-03-04T16:37:40.863 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.864 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.865 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004204e1df94fff36b-0065e5f8d4-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.866 DEBUG:boto:path=/sbnvju-26/obj 2024-03-04T16:37:40.866 DEBUG:boto:auth_path=/sbnvju-26/obj 2024-03-04T16:37:40.866 DEBUG:boto:Method: HEAD 2024-03-04T16:37:40.866 DEBUG:boto:Path: /sbnvju-26/obj 2024-03-04T16:37:40.866 DEBUG:boto:Data: 2024-03-04T16:37:40.866 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.866 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.866 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.866 DEBUG:boto:Params: {} 2024-03-04T16:37:40.866 DEBUG:boto:Token: None 2024-03-04T16:37:40.866 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-26/obj 2024-03-04T16:37:40.867 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4= 2024-03-04T16:37:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sHK+lV41pqRwIFUsDdRpzi6Wnx4=', 'Content-Length': '0'} 2024-03-04T16:37:40.868 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000045cb962e5ab4417b-0065e5f8d4-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-26 zones={test-zone3, test-zone2} 2024-03-04T16:37:40.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-03-04T16:37:40.870 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-27 2024-03-04T16:37:40.870 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:37:40.871 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:37:40.871 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.871 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:37:40.871 DEBUG:boto:Data: 2024-03-04T16:37:40.871 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.871 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.871 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.871 DEBUG:boto:Params: {} 2024-03-04T16:37:40.871 DEBUG:boto:Token: None 2024-03-04T16:37:40.871 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-27/ 2024-03-04T16:37:40.872 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AEVUP3RAnB7bRpIv7Cegw1nsDfo= 2024-03-04T16:37:40.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AEVUP3RAnB7bRpIv7Cegw1nsDfo=', 'Content-Length': '0'} 2024-03-04T16:37:40.885 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9df5ac0c6046ec2-0065e5f8d4-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.885 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-28 2024-03-04T16:37:40.885 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:37:40.885 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:37:40.886 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.886 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:37:40.886 DEBUG:boto:Data: 2024-03-04T16:37:40.886 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.886 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.886 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.886 DEBUG:boto:Params: {} 2024-03-04T16:37:40.886 DEBUG:boto:Token: None 2024-03-04T16:37:40.886 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-28/ 2024-03-04T16:37:40.886 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hmduYqdwbHOIp+OHLqDApks1VcU= 2024-03-04T16:37:40.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hmduYqdwbHOIp+OHLqDApks1VcU=', 'Content-Length': '0'} 2024-03-04T16:37:40.910 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c7cc8e5d6e18c6b-0065e5f8d4-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.910 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-29 2024-03-04T16:37:40.910 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:37:40.910 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:37:40.910 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.910 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:37:40.910 DEBUG:boto:Data: 2024-03-04T16:37:40.910 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.910 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.910 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.911 DEBUG:boto:Params: {} 2024-03-04T16:37:40.911 DEBUG:boto:Token: None 2024-03-04T16:37:40.911 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-29/ 2024-03-04T16:37:40.911 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8lSMJ7+LlJoHj4w1PGGhZO2i8is= 2024-03-04T16:37:40.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8lSMJ7+LlJoHj4w1PGGhZO2i8is=', 'Content-Length': '0'} 2024-03-04T16:37:40.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a219b2cf81f091f1-0065e5f8d4-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.932 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:37:40.933 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:37:40.933 DEBUG:boto:path=/sbnvju-27/?versioning 2024-03-04T16:37:40.933 DEBUG:boto:auth_path=/sbnvju-27/?versioning 2024-03-04T16:37:40.933 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.933 DEBUG:boto:Path: /sbnvju-27/?versioning 2024-03-04T16:37:40.933 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:37:40.933 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.933 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.933 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.933 DEBUG:boto:Params: {} 2024-03-04T16:37:40.934 DEBUG:boto:Token: None 2024-03-04T16:37:40.934 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-27/?versioning 2024-03-04T16:37:40.934 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FXoA661WLDrbulG9W96L2pdDLYs= 2024-03-04T16:37:40.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FXoA661WLDrbulG9W96L2pdDLYs=', 'Content-Length': '228'} 2024-03-04T16:37:40.939 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056a632d89810db8e-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.939 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:37:40.939 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:37:40.939 DEBUG:boto:path=/sbnvju-27/?versioning 2024-03-04T16:37:40.939 DEBUG:boto:auth_path=/sbnvju-27/?versioning 2024-03-04T16:37:40.939 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.939 DEBUG:boto:Path: /sbnvju-27/?versioning 2024-03-04T16:37:40.939 DEBUG:boto:Data: Suspended Disabled 2024-03-04T16:37:40.940 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.940 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:40.940 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.940 DEBUG:boto:Params: {} 2024-03-04T16:37:40.940 DEBUG:boto:Token: None 2024-03-04T16:37:40.940 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-27/?versioning 2024-03-04T16:37:40.940 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:FXoA661WLDrbulG9W96L2pdDLYs= 2024-03-04T16:37:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FXoA661WLDrbulG9W96L2pdDLYs=', 'Content-Length': '230'} 2024-03-04T16:37:40.945 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff8a363917b6f160-0065e5f8d4-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.945 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:37:40.945 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:37:40.945 DEBUG:boto:path=/sbnvju-28/?versioning 2024-03-04T16:37:40.945 DEBUG:boto:auth_path=/sbnvju-28/?versioning 2024-03-04T16:37:40.945 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.945 DEBUG:boto:Path: /sbnvju-28/?versioning 2024-03-04T16:37:40.945 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:37:40.946 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.946 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.946 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.946 DEBUG:boto:Params: {} 2024-03-04T16:37:40.946 DEBUG:boto:Token: None 2024-03-04T16:37:40.946 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-28/?versioning 2024-03-04T16:37:40.946 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KzZ40/KXkf6QA61/W0zlTHvm1e4= 2024-03-04T16:37:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KzZ40/KXkf6QA61/W0zlTHvm1e4=', 'Content-Length': '228'} 2024-03-04T16:37:40.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d890235ddbc6c1cc-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.954 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:37:40.954 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:37:40.954 DEBUG:boto:path=/sbnvju-28/?versioning 2024-03-04T16:37:40.954 DEBUG:boto:auth_path=/sbnvju-28/?versioning 2024-03-04T16:37:40.954 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.954 DEBUG:boto:Path: /sbnvju-28/?versioning 2024-03-04T16:37:40.954 DEBUG:boto:Data: Suspended Disabled 2024-03-04T16:37:40.955 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.955 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:40.955 DEBUG:boto:Port: 8000 2024-03-04T16:37:40.955 DEBUG:boto:Params: {} 2024-03-04T16:37:40.955 DEBUG:boto:Token: None 2024-03-04T16:37:40.955 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-28/?versioning 2024-03-04T16:37:40.955 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KzZ40/KXkf6QA61/W0zlTHvm1e4= 2024-03-04T16:37:40.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KzZ40/KXkf6QA61/W0zlTHvm1e4=', 'Content-Length': '230'} 2024-03-04T16:37:40.962 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000988cf746bb4b378b-0065e5f8d4-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.962 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:37:40.963 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:37:40.963 DEBUG:boto:path=/sbnvju-29/?versioning 2024-03-04T16:37:40.963 DEBUG:boto:auth_path=/sbnvju-29/?versioning 2024-03-04T16:37:40.963 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.963 DEBUG:boto:Path: /sbnvju-29/?versioning 2024-03-04T16:37:40.963 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:37:40.963 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.963 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.963 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.963 DEBUG:boto:Params: {} 2024-03-04T16:37:40.963 DEBUG:boto:Token: None 2024-03-04T16:37:40.964 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-29/?versioning 2024-03-04T16:37:40.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wjro2TWo1xgcT2sEqYbEyHzLugQ= 2024-03-04T16:37:40.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wjro2TWo1xgcT2sEqYbEyHzLugQ=', 'Content-Length': '228'} 2024-03-04T16:37:40.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000353a9e7a620b38ce-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.971 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:37:40.971 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:37:40.971 DEBUG:boto:path=/sbnvju-29/?versioning 2024-03-04T16:37:40.971 DEBUG:boto:auth_path=/sbnvju-29/?versioning 2024-03-04T16:37:40.971 DEBUG:boto:Method: PUT 2024-03-04T16:37:40.971 DEBUG:boto:Path: /sbnvju-29/?versioning 2024-03-04T16:37:40.971 DEBUG:boto:Data: Suspended Disabled 2024-03-04T16:37:40.971 DEBUG:boto:Headers: {} 2024-03-04T16:37:40.972 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:40.972 DEBUG:boto:Port: 8001 2024-03-04T16:37:40.972 DEBUG:boto:Params: {} 2024-03-04T16:37:40.972 DEBUG:boto:Token: None 2024-03-04T16:37:40.972 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:37:40 GMT /sbnvju-29/?versioning 2024-03-04T16:37:40.972 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wjro2TWo1xgcT2sEqYbEyHzLugQ= 2024-03-04T16:37:40.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wjro2TWo1xgcT2sEqYbEyHzLugQ=', 'Content-Length': '230'} 2024-03-04T16:37:40.979 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000080195dd2025b2be-0065e5f8d4-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:40.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:40.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:40.980 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:37:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:41.090 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:37:41.110 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570260.990690_383.1","last_update":"2024-03-04T16:37:40.990690Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:37:41.111 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:41.111 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:37:41.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:41.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:41.111 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:37:41.157 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:41.157 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:41.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:41.214+0000 7f2575411800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:41.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:41.215+0000 7f2575411800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:41.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.910810_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.910810Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:41Z"} 2024-03-04T16:37:41.221 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.910810_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.910810Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:41Z"} 2024-03-04T16:37:41.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570260.910810_365.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:37:40.910810Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:37:41.221 DEBUG:rgw_multi.tests:log_status={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:41.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570260.910810_365.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:41.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570260.990690_383.1 target=1_1709570260.910810_365.1 2024-03-04T16:37:46.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:46.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:46.224 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:37:46.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:46.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:46.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:46.339+0000 7fe94bf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:46.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:46.340+0000 7fe94bf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:46.347 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.990690_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.990690Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:46Z"} 2024-03-04T16:37:46.347 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.990690_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.990690Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:46Z"} 2024-03-04T16:37:46.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570260.990690_383.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:37:40.990690Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:37:46.347 DEBUG:rgw_multi.tests:log_status={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:46.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:46.347 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-03-04T16:37:46.348 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-03-04T16:37:46.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:46.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:37:46.348 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:37:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:46.467+0000 7fd36ab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:37:46.469+0000 7fd36ab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:37:46.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.990690_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.990690Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:46Z"} 2024-03-04T16:37:46.477 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570260.990690_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:37:40.990690Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:37:46Z"} 2024-03-04T16:37:46.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570260.990690_383.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:37:40.990690Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:37:46.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:46.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570260.990690_383.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:37:46.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-03-04T16:37:46.478 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:37:46.478 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:37:46.478 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.478 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:37:46.478 DEBUG:boto:Data: 2024-03-04T16:37:46.478 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.478 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:46.478 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.478 DEBUG:boto:Params: {} 2024-03-04T16:37:46.478 DEBUG:boto:Token: None 2024-03-04T16:37:46.479 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-27/ 2024-03-04T16:37:46.479 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Elw15Zd6wted6Q68DhzpqmkEQq0= 2024-03-04T16:37:46.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Elw15Zd6wted6Q68DhzpqmkEQq0=', 'Content-Length': '0'} 2024-03-04T16:37:46.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a007c07c39c4ea2-0065e5f8da-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.481 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:37:46.481 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:37:46.482 DEBUG:boto:Method: PUT 2024-03-04T16:37:46.482 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:37:46.482 DEBUG:boto:Data: 2024-03-04T16:37:46.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:37:46.482 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:46.482 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.482 DEBUG:boto:Params: {} 2024-03-04T16:37:46.482 DEBUG:boto:Token: None 2024-03-04T16:37:46.482 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-27/obj 2024-03-04T16:37:46.482 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5asGFIxkCx3UcwJ73TQkul7lkDk= 2024-03-04T16:37:46.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5asGFIxkCx3UcwJ73TQkul7lkDk='} 2024-03-04T16:37:46.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed967543ab8335d9-0065e5f8da-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.524 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:37:46.524 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:37:46.524 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.524 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:37:46.525 DEBUG:boto:Data: 2024-03-04T16:37:46.525 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.525 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:46.525 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.525 DEBUG:boto:Params: {} 2024-03-04T16:37:46.525 DEBUG:boto:Token: None 2024-03-04T16:37:46.525 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-27/ 2024-03-04T16:37:46.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Elw15Zd6wted6Q68DhzpqmkEQq0= 2024-03-04T16:37:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Elw15Zd6wted6Q68DhzpqmkEQq0=', 'Content-Length': '0'} 2024-03-04T16:37:46.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004e26d8fc78460a3-0065e5f8da-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.528 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:37:46.528 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:37:46.528 DEBUG:boto:Method: DELETE 2024-03-04T16:37:46.528 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:37:46.528 DEBUG:boto:Data: 2024-03-04T16:37:46.528 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.528 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:37:46.528 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.528 DEBUG:boto:Params: {} 2024-03-04T16:37:46.528 DEBUG:boto:Token: None 2024-03-04T16:37:46.529 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-27/obj 2024-03-04T16:37:46.529 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:O4qNo4UVzEayfMcsD8fnCrQ1Vds= 2024-03-04T16:37:46.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4qNo4UVzEayfMcsD8fnCrQ1Vds=', 'Content-Length': '0'} 2024-03-04T16:37:46.537 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000002674dea16bb5a9ed-0065e5f8da-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.538 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:37:46.538 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:37:46.538 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.538 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:37:46.538 DEBUG:boto:Data: 2024-03-04T16:37:46.538 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.538 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:46.538 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.538 DEBUG:boto:Params: {} 2024-03-04T16:37:46.539 DEBUG:boto:Token: None 2024-03-04T16:37:46.539 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-28/ 2024-03-04T16:37:46.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3eTXchypJpaQPHP9e+UST8KXVpg= 2024-03-04T16:37:46.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3eTXchypJpaQPHP9e+UST8KXVpg=', 'Content-Length': '0'} 2024-03-04T16:37:46.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d188bbad09932e6c-0065e5f8da-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.541 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:37:46.542 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:37:46.542 DEBUG:boto:Method: PUT 2024-03-04T16:37:46.542 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:37:46.542 DEBUG:boto:Data: 2024-03-04T16:37:46.542 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:37:46.542 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:46.542 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.542 DEBUG:boto:Params: {} 2024-03-04T16:37:46.542 DEBUG:boto:Token: None 2024-03-04T16:37:46.542 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-28/obj 2024-03-04T16:37:46.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KH2qvFdm6bqyNc6do1SSzRrQlxU= 2024-03-04T16:37:46.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KH2qvFdm6bqyNc6do1SSzRrQlxU='} 2024-03-04T16:37:46.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000160ab1e2d81eece0-0065e5f8da-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.548 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:37:46.548 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:37:46.548 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.549 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:37:46.549 DEBUG:boto:Data: 2024-03-04T16:37:46.549 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.549 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:46.549 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.549 DEBUG:boto:Params: {} 2024-03-04T16:37:46.549 DEBUG:boto:Token: None 2024-03-04T16:37:46.549 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-28/ 2024-03-04T16:37:46.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3eTXchypJpaQPHP9e+UST8KXVpg= 2024-03-04T16:37:46.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3eTXchypJpaQPHP9e+UST8KXVpg=', 'Content-Length': '0'} 2024-03-04T16:37:46.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0ab0cbd18f92809-0065e5f8da-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.551 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:37:46.551 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:37:46.552 DEBUG:boto:Method: DELETE 2024-03-04T16:37:46.552 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:37:46.552 DEBUG:boto:Data: 2024-03-04T16:37:46.552 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.552 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:37:46.552 DEBUG:boto:Port: 8000 2024-03-04T16:37:46.552 DEBUG:boto:Params: {} 2024-03-04T16:37:46.552 DEBUG:boto:Token: None 2024-03-04T16:37:46.552 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-28/obj 2024-03-04T16:37:46.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:JBw2QKqjssvXDc4SjlS2msWKHq0= 2024-03-04T16:37:46.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JBw2QKqjssvXDc4SjlS2msWKHq0=', 'Content-Length': '0'} 2024-03-04T16:37:46.561 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000000e07fde9a53e3209-0065e5f8da-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.561 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:37:46.562 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:37:46.562 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.562 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:37:46.562 DEBUG:boto:Data: 2024-03-04T16:37:46.562 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:46.562 DEBUG:boto:Port: 8001 2024-03-04T16:37:46.562 DEBUG:boto:Params: {} 2024-03-04T16:37:46.562 DEBUG:boto:Token: None 2024-03-04T16:37:46.562 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-29/ 2024-03-04T16:37:46.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vvCn91w3BLhE92HFGzd6sKX9sT8= 2024-03-04T16:37:46.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vvCn91w3BLhE92HFGzd6sKX9sT8=', 'Content-Length': '0'} 2024-03-04T16:37:46.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b3e304372ad3e13-0065e5f8da-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.565 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:37:46.565 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:37:46.565 DEBUG:boto:Method: PUT 2024-03-04T16:37:46.565 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:37:46.565 DEBUG:boto:Data: 2024-03-04T16:37:46.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-03-04T16:37:46.565 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:46.566 DEBUG:boto:Port: 8001 2024-03-04T16:37:46.566 DEBUG:boto:Params: {} 2024-03-04T16:37:46.566 DEBUG:boto:Token: None 2024-03-04T16:37:46.566 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-29/obj 2024-03-04T16:37:46.566 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:9C640xeKc1OGPxNHKWG9YvnbXHU= 2024-03-04T16:37:46.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9C640xeKc1OGPxNHKWG9YvnbXHU='} 2024-03-04T16:37:46.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d16a365de2893877-0065e5f8da-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.572 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:37:46.572 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:37:46.572 DEBUG:boto:Method: HEAD 2024-03-04T16:37:46.572 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:37:46.572 DEBUG:boto:Data: 2024-03-04T16:37:46.572 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.572 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:46.572 DEBUG:boto:Port: 8001 2024-03-04T16:37:46.572 DEBUG:boto:Params: {} 2024-03-04T16:37:46.572 DEBUG:boto:Token: None 2024-03-04T16:37:46.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-29/ 2024-03-04T16:37:46.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vvCn91w3BLhE92HFGzd6sKX9sT8= 2024-03-04T16:37:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vvCn91w3BLhE92HFGzd6sKX9sT8=', 'Content-Length': '0'} 2024-03-04T16:37:46.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b501d501c28a5f6f-0065e5f8da-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.575 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:37:46.575 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:37:46.575 DEBUG:boto:Method: DELETE 2024-03-04T16:37:46.575 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:37:46.575 DEBUG:boto:Data: 2024-03-04T16:37:46.575 DEBUG:boto:Headers: {} 2024-03-04T16:37:46.575 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:37:46.575 DEBUG:boto:Port: 8001 2024-03-04T16:37:46.575 DEBUG:boto:Params: {} 2024-03-04T16:37:46.576 DEBUG:boto:Token: None 2024-03-04T16:37:46.576 DEBUG:boto:StringToSign: DELETE Mon, 04 Mar 2024 16:37:46 GMT /sbnvju-29/obj 2024-03-04T16:37:46.576 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:w11IZ1XYG+Av1RHb2FMvIGqHfzE= 2024-03-04T16:37:46.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:37:46 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:w11IZ1XYG+Av1RHb2FMvIGqHfzE=', 'Content-Length': '0'} 2024-03-04T16:37:46.584 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000ce854b88e83254ed-0065e5f8da-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:37:46 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:37:46.584 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-27 2024-03-04T16:37:46.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:46.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:37:46.585 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:37:46.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:37:46.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:37:46.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:46.699+0000 7f3a2a867800 1 waiting to reach incremental sync.. 2024-03-04T16:37:51.686 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:51.699+0000 7f3a2a867800 1 waiting to reach incremental sync.. 2024-03-04T16:37:56.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:37:56.699+0000 7f3a2a867800 1 waiting to reach incremental sync.. 2024-03-04T16:38:01.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:01.701+0000 7f3a2a867800 1 bucket sync caught up with source: 2024-03-04T16:38:01.689 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.30.7, , ] 2024-03-04T16:38:01.689 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.30.7, , ] 2024-03-04T16:38:01.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:01.701+0000 7f3a2a867800 0 bucket checkpoint complete 2024-03-04T16:38:01.713 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-27 2024-03-04T16:38:01.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:01.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:01.713 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:01.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:01.846+0000 7f12c062e800 1 waiting to reach incremental sync.. 2024-03-04T16:38:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:06.849+0000 7f12c062e800 1 bucket sync caught up with source: 2024-03-04T16:38:06.836 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.30.7, , ] 2024-03-04T16:38:06.836 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.30.7, , ] 2024-03-04T16:38:06.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:06.849+0000 7f12c062e800 0 bucket checkpoint complete 2024-03-04T16:38:06.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-27 2024-03-04T16:38:06.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:06.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:06.860 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:06.986+0000 7fe900fa2800 1 waiting to reach incremental sync.. 2024-03-04T16:38:11.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:11.986+0000 7fe900fa2800 1 waiting to reach incremental sync.. 2024-03-04T16:38:16.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:16.987+0000 7fe900fa2800 1 waiting to reach incremental sync.. 2024-03-04T16:38:21.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:21.989+0000 7fe900fa2800 1 bucket sync caught up with source: 2024-03-04T16:38:21.976 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.24.14, , ] 2024-03-04T16:38:21.977 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.24.14, , ] 2024-03-04T16:38:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:21.989+0000 7fe900fa2800 0 bucket checkpoint complete 2024-03-04T16:38:22.001 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-27 2024-03-04T16:38:22.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.003 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:22.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.132+0000 7f4b8649d800 1 bucket sync caught up with source: 2024-03-04T16:38:22.119 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.24.14, , ] 2024-03-04T16:38:22.119 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.24.14, , ] 2024-03-04T16:38:22.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.132+0000 7f4b8649d800 0 bucket checkpoint complete 2024-03-04T16:38:22.138 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-27 2024-03-04T16:38:22.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.139 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.196 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.196 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.260+0000 7f074e8c0800 1 bucket sync caught up with source: 2024-03-04T16:38:22.247 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.22.14, , ] 2024-03-04T16:38:22.247 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.22.14, , ] 2024-03-04T16:38:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.260+0000 7f074e8c0800 0 bucket checkpoint complete 2024-03-04T16:38:22.266 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-27 2024-03-04T16:38:22.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.266 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:22.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:22.395+0000 7fdc23e32800 1 bucket sync caught up with source: 2024-03-04T16:38:22.382 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.22.14, , ] 2024-03-04T16:38:22.382 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.22.14, , ] 2024-03-04T16:38:22.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:22.395+0000 7fdc23e32800 0 bucket checkpoint complete 2024-03-04T16:38:22.401 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-27 zones={test-zone2, test-zone1} 2024-03-04T16:38:22.401 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.401 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.401 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.401 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.401 DEBUG:boto:Data: 2024-03-04T16:38:22.401 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.402 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.402 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.402 DEBUG:boto:Params: {} 2024-03-04T16:38:22.402 DEBUG:boto:Token: None 2024-03-04T16:38:22.402 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.402 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bad191a73e1f9dbc-0065e5f8fe-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.405 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.405 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.405 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.405 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.406 DEBUG:boto:Data: 2024-03-04T16:38:22.406 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.406 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.406 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.406 DEBUG:boto:Params: {} 2024-03-04T16:38:22.406 DEBUG:boto:Token: None 2024-03-04T16:38:22.406 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.406 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e67ca60e6929110-0065e5f8fe-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.409 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:22.409 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.409 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.409 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.409 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.409 DEBUG:boto:Method: GET 2024-03-04T16:38:22.409 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.409 DEBUG:boto:Data: 2024-03-04T16:38:22.410 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.410 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.410 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.410 DEBUG:boto:Params: {} 2024-03-04T16:38:22.410 DEBUG:boto:Token: None 2024-03-04T16:38:22.410 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.410 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d59960c73ae76ed-0065e5f8fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.460 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.461 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:22.461 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.461 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.461 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.461 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.461 DEBUG:boto:Method: GET 2024-03-04T16:38:22.462 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.462 DEBUG:boto:Data: 2024-03-04T16:38:22.462 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.462 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.462 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.462 DEBUG:boto:Params: {} 2024-03-04T16:38:22.462 DEBUG:boto:Token: None 2024-03-04T16:38:22.462 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.462 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a15bfcc8b8ce02-0065e5f8fe-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.465 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.465 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.465 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.465 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.465 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.466 DEBUG:boto:Method: GET 2024-03-04T16:38:22.466 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.466 DEBUG:boto:Data: 2024-03-04T16:38:22.466 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.466 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.466 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.466 DEBUG:boto:Params: {} 2024-03-04T16:38:22.466 DEBUG:boto:Token: None 2024-03-04T16:38:22.466 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.467 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8cad7a2b6fe8f73-0065e5f8fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.469 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.469 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.470 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.470 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.470 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.470 DEBUG:boto:Method: GET 2024-03-04T16:38:22.470 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.470 DEBUG:boto:Data: 2024-03-04T16:38:22.470 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.470 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.470 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.470 DEBUG:boto:Params: {} 2024-03-04T16:38:22.470 DEBUG:boto:Token: None 2024-03-04T16:38:22.471 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.471 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087477eacb07f39aa-0065e5f8fe-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.473 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:22.473 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.474 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.474 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.474 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.474 DEBUG:boto:Data: 2024-03-04T16:38:22.474 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.474 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.474 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.474 DEBUG:boto:Params: {} 2024-03-04T16:38:22.474 DEBUG:boto:Token: None 2024-03-04T16:38:22.474 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.475 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.476 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000067b060a33a710ab7-0065e5f8fe-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.476 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.476 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.476 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.476 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.476 DEBUG:boto:Data: 2024-03-04T16:38:22.477 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.477 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.477 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.477 DEBUG:boto:Params: {} 2024-03-04T16:38:22.477 DEBUG:boto:Token: None 2024-03-04T16:38:22.477 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.477 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.479 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006409efa100805665-0065e5f8fe-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-27 zones={test-zone2, test-zone1} 2024-03-04T16:38:22.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-27 zones={test-zone3, test-zone1} 2024-03-04T16:38:22.479 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.479 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.479 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.479 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.479 DEBUG:boto:Data: 2024-03-04T16:38:22.480 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.480 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.480 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.480 DEBUG:boto:Params: {} 2024-03-04T16:38:22.480 DEBUG:boto:Token: None 2024-03-04T16:38:22.480 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.480 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000538b02f0c8b188f4-0065e5f8fe-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.482 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.483 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.483 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.483 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.483 DEBUG:boto:Data: 2024-03-04T16:38:22.483 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.483 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.483 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.483 DEBUG:boto:Params: {} 2024-03-04T16:38:22.483 DEBUG:boto:Token: None 2024-03-04T16:38:22.483 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.483 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c664bdb2e7d53f-0065e5f8fe-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:22.486 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.486 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.486 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.486 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.486 DEBUG:boto:Method: GET 2024-03-04T16:38:22.486 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.486 DEBUG:boto:Data: 2024-03-04T16:38:22.486 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.486 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.486 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.487 DEBUG:boto:Params: {} 2024-03-04T16:38:22.487 DEBUG:boto:Token: None 2024-03-04T16:38:22.487 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.487 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008009dff81078f490-0065e5f8fe-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.490 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:22.490 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.490 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.490 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.490 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.490 DEBUG:boto:Method: GET 2024-03-04T16:38:22.491 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.491 DEBUG:boto:Data: 2024-03-04T16:38:22.491 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.491 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.491 DEBUG:boto:Params: {} 2024-03-04T16:38:22.491 DEBUG:boto:Token: None 2024-03-04T16:38:22.491 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.491 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cda13f8df4256c75-0065e5f8fe-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.540 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.541 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.541 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.541 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.541 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.541 DEBUG:boto:Method: GET 2024-03-04T16:38:22.541 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.541 DEBUG:boto:Data: 2024-03-04T16:38:22.541 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.541 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.541 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.541 DEBUG:boto:Params: {} 2024-03-04T16:38:22.542 DEBUG:boto:Token: None 2024-03-04T16:38:22.542 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.542 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052167e6d3d33eef3-0065e5f8fe-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.544 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.545 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.545 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.545 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.545 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.545 DEBUG:boto:Method: GET 2024-03-04T16:38:22.545 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.545 DEBUG:boto:Data: 2024-03-04T16:38:22.545 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.545 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.546 DEBUG:boto:Params: {} 2024-03-04T16:38:22.546 DEBUG:boto:Token: None 2024-03-04T16:38:22.546 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e819b78843abe7-0065e5f8fe-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.548 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:22.549 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.549 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.549 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.549 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.549 DEBUG:boto:Data: 2024-03-04T16:38:22.549 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.549 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.550 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.550 DEBUG:boto:Params: {} 2024-03-04T16:38:22.550 DEBUG:boto:Token: None 2024-03-04T16:38:22.550 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.550 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.551 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009e031412c02bf41d-0065e5f8fe-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.552 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.552 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.552 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.552 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.552 DEBUG:boto:Data: 2024-03-04T16:38:22.552 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.552 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:22.552 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.552 DEBUG:boto:Params: {} 2024-03-04T16:38:22.553 DEBUG:boto:Token: None 2024-03-04T16:38:22.553 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.553 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.554 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000180eef5e98cee724-0065e5f8fe-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-27 zones={test-zone3, test-zone1} 2024-03-04T16:38:22.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-27 zones={test-zone3, test-zone2} 2024-03-04T16:38:22.555 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.555 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.555 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.555 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.555 DEBUG:boto:Data: 2024-03-04T16:38:22.555 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.555 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.555 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.555 DEBUG:boto:Params: {} 2024-03-04T16:38:22.555 DEBUG:boto:Token: None 2024-03-04T16:38:22.556 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.556 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b747b219aadf92b4-0065e5f8fe-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.558 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.558 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.558 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.558 DEBUG:boto:Path: /sbnvju-27/ 2024-03-04T16:38:22.558 DEBUG:boto:Data: 2024-03-04T16:38:22.558 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.558 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.559 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.559 DEBUG:boto:Params: {} 2024-03-04T16:38:22.559 DEBUG:boto:Token: None 2024-03-04T16:38:22.559 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/ 2024-03-04T16:38:22.559 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y= 2024-03-04T16:38:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qug5Ttm9eDeobBlGKxKPLIjAE2Y=', 'Content-Length': '0'} 2024-03-04T16:38:22.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001390096adc74cfa5-0065e5f8fe-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:22.562 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.562 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.562 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.562 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.562 DEBUG:boto:Method: GET 2024-03-04T16:38:22.562 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.562 DEBUG:boto:Data: 2024-03-04T16:38:22.562 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.562 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.562 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.562 DEBUG:boto:Params: {} 2024-03-04T16:38:22.563 DEBUG:boto:Token: None 2024-03-04T16:38:22.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.563 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099a75b888e39b81b-0065e5f8fe-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.608 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.609 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:22.609 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.609 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.609 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.609 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.609 DEBUG:boto:Method: GET 2024-03-04T16:38:22.609 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.609 DEBUG:boto:Data: 2024-03-04T16:38:22.609 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.609 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.609 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.610 DEBUG:boto:Params: {} 2024-03-04T16:38:22.610 DEBUG:boto:Token: None 2024-03-04T16:38:22.610 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.610 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000092e54935cac3834-0065e5f8fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.612 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:22.613 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.613 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.613 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.614 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.614 DEBUG:boto:Method: GET 2024-03-04T16:38:22.614 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.614 DEBUG:boto:Data: 2024-03-04T16:38:22.614 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.614 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.614 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.614 DEBUG:boto:Params: {} 2024-03-04T16:38:22.614 DEBUG:boto:Token: None 2024-03-04T16:38:22.615 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.615 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036603709f3fac1f6-0065e5f8fe-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.618 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.618 DEBUG:boto:path=/sbnvju-27/ 2024-03-04T16:38:22.618 DEBUG:boto:auth_path=/sbnvju-27/ 2024-03-04T16:38:22.618 DEBUG:boto:path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.618 DEBUG:boto:auth_path=/sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.618 DEBUG:boto:Method: GET 2024-03-04T16:38:22.618 DEBUG:boto:Path: /sbnvju-27/?versions&max-keys=999 2024-03-04T16:38:22.619 DEBUG:boto:Data: 2024-03-04T16:38:22.619 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.619 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.619 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.619 DEBUG:boto:Params: {} 2024-03-04T16:38:22.619 DEBUG:boto:Token: None 2024-03-04T16:38:22.619 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/?versions 2024-03-04T16:38:22.619 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk= 2024-03-04T16:38:22.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EMC1RJyT4DjRsl7FxW5SCrVW0yk=', 'Content-Length': '0'} 2024-03-04T16:38:22.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006b290a74fa3cf7-0065e5f8fe-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.664 DEBUG:boto:b'sbnvju-27999falseobjnulltrue2024-03-04T16:37:46.544Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:22.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:22.665 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.665 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.665 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.665 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.665 DEBUG:boto:Data: 2024-03-04T16:38:22.665 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.665 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:22.665 DEBUG:boto:Port: 8001 2024-03-04T16:38:22.666 DEBUG:boto:Params: {} 2024-03-04T16:38:22.666 DEBUG:boto:Token: None 2024-03-04T16:38:22.666 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.666 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.668 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002160cd70af1c998d-0065e5f8fe-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.668 DEBUG:boto:path=/sbnvju-27/obj 2024-03-04T16:38:22.668 DEBUG:boto:auth_path=/sbnvju-27/obj 2024-03-04T16:38:22.668 DEBUG:boto:Method: HEAD 2024-03-04T16:38:22.668 DEBUG:boto:Path: /sbnvju-27/obj 2024-03-04T16:38:22.669 DEBUG:boto:Data: 2024-03-04T16:38:22.669 DEBUG:boto:Headers: {} 2024-03-04T16:38:22.669 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:22.669 DEBUG:boto:Port: 8000 2024-03-04T16:38:22.669 DEBUG:boto:Params: {} 2024-03-04T16:38:22.669 DEBUG:boto:Token: None 2024-03-04T16:38:22.669 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:22 GMT /sbnvju-27/obj 2024-03-04T16:38:22.669 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q= 2024-03-04T16:38:22.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:22 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:8+8A2Ex6mcJr68sdPPwTJp4X+7Q=', 'Content-Length': '0'} 2024-03-04T16:38:22.671 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000eb70cc2ec71a9493-0065e5f8fe-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:22 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:22.671 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-27 zones={test-zone3, test-zone2} 2024-03-04T16:38:22.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-28 2024-03-04T16:38:22.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.671 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:22.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:22.790+0000 7f328ed30800 1 bucket sync caught up with source: 2024-03-04T16:38:22.778 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.75.14, , ] 2024-03-04T16:38:22.778 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.75.14, , ] 2024-03-04T16:38:22.778 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:22.790+0000 7f328ed30800 0 bucket checkpoint complete 2024-03-04T16:38:22.796 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-28 2024-03-04T16:38:22.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.796 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.916+0000 7fc3b1c0f800 1 bucket sync caught up with source: 2024-03-04T16:38:22.903 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.75.14, , ] 2024-03-04T16:38:22.904 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.75.14, , ] 2024-03-04T16:38:22.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:22.916+0000 7fc3b1c0f800 0 bucket checkpoint complete 2024-03-04T16:38:22.923 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-28 2024-03-04T16:38:22.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:22.924 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.043+0000 7f5069a1e800 1 bucket sync caught up with source: 2024-03-04T16:38:23.031 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.85.7, , ] 2024-03-04T16:38:23.031 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.85.7, , ] 2024-03-04T16:38:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.043+0000 7f5069a1e800 0 bucket checkpoint complete 2024-03-04T16:38:23.050 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-28 2024-03-04T16:38:23.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.050 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:23.105 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.168+0000 7f750d08c800 1 bucket sync caught up with source: 2024-03-04T16:38:23.155 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.85.7, , ] 2024-03-04T16:38:23.156 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.85.7, , ] 2024-03-04T16:38:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.168+0000 7f750d08c800 0 bucket checkpoint complete 2024-03-04T16:38:23.174 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-28 2024-03-04T16:38:23.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.175 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.297+0000 7f70ecf1e800 1 bucket sync caught up with source: 2024-03-04T16:38:23.284 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.107.14, , ] 2024-03-04T16:38:23.284 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.107.14, , ] 2024-03-04T16:38:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.297+0000 7f70ecf1e800 0 bucket checkpoint complete 2024-03-04T16:38:23.303 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-28 2024-03-04T16:38:23.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.304 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:23.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:23.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:23.428+0000 7f9164b3c800 1 bucket sync caught up with source: 2024-03-04T16:38:23.415 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.107.14, , ] 2024-03-04T16:38:23.415 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.107.14, , ] 2024-03-04T16:38:23.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:23.428+0000 7f9164b3c800 0 bucket checkpoint complete 2024-03-04T16:38:23.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-28 zones={test-zone2, test-zone1} 2024-03-04T16:38:23.433 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.433 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.433 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.433 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.433 DEBUG:boto:Data: 2024-03-04T16:38:23.434 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.434 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.434 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.434 DEBUG:boto:Params: {} 2024-03-04T16:38:23.434 DEBUG:boto:Token: None 2024-03-04T16:38:23.434 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.434 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb62e08ec381ea54-0065e5f8ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.437 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.437 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.437 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.437 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.437 DEBUG:boto:Data: 2024-03-04T16:38:23.437 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.437 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.437 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.437 DEBUG:boto:Params: {} 2024-03-04T16:38:23.438 DEBUG:boto:Token: None 2024-03-04T16:38:23.438 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.438 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a3cd580b6d81d72-0065e5f8ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:23.440 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.440 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.440 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.440 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.440 DEBUG:boto:Method: GET 2024-03-04T16:38:23.441 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.441 DEBUG:boto:Data: 2024-03-04T16:38:23.441 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.441 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.441 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.441 DEBUG:boto:Params: {} 2024-03-04T16:38:23.441 DEBUG:boto:Token: None 2024-03-04T16:38:23.441 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.441 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4371df8cc48e19d-0065e5f8ff-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.443 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.444 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:23.444 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.444 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.444 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.444 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.444 DEBUG:boto:Method: GET 2024-03-04T16:38:23.445 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.445 DEBUG:boto:Data: 2024-03-04T16:38:23.445 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.445 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.445 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.447 DEBUG:boto:Params: {} 2024-03-04T16:38:23.447 DEBUG:boto:Token: None 2024-03-04T16:38:23.447 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.447 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000762d39034d4106f9-0065e5f8ff-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.449 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.450 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.450 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.450 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.450 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.450 DEBUG:boto:Method: GET 2024-03-04T16:38:23.450 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.450 DEBUG:boto:Data: 2024-03-04T16:38:23.450 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.450 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.450 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.451 DEBUG:boto:Params: {} 2024-03-04T16:38:23.451 DEBUG:boto:Token: None 2024-03-04T16:38:23.451 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.451 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000152170f1387691a2-0065e5f8ff-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.453 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.454 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.454 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.454 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.454 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.454 DEBUG:boto:Method: GET 2024-03-04T16:38:23.454 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.454 DEBUG:boto:Data: 2024-03-04T16:38:23.454 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.454 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.454 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.455 DEBUG:boto:Params: {} 2024-03-04T16:38:23.455 DEBUG:boto:Token: None 2024-03-04T16:38:23.455 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.455 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002576e48c2b525e31-0065e5f8ff-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.500 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:23.501 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.501 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.501 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.501 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.501 DEBUG:boto:Data: 2024-03-04T16:38:23.501 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.501 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.501 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.501 DEBUG:boto:Params: {} 2024-03-04T16:38:23.502 DEBUG:boto:Token: None 2024-03-04T16:38:23.502 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.504 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006fc691bc6bfb50e2-0065e5f8ff-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.504 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.504 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.504 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.504 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.504 DEBUG:boto:Data: 2024-03-04T16:38:23.504 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.504 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.504 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.504 DEBUG:boto:Params: {} 2024-03-04T16:38:23.505 DEBUG:boto:Token: None 2024-03-04T16:38:23.505 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.505 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.506 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002c1a8c29d566b2ad-0065e5f8ff-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-28 zones={test-zone2, test-zone1} 2024-03-04T16:38:23.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-28 zones={test-zone3, test-zone1} 2024-03-04T16:38:23.507 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.507 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.507 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.507 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.507 DEBUG:boto:Data: 2024-03-04T16:38:23.507 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.507 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.507 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.507 DEBUG:boto:Params: {} 2024-03-04T16:38:23.507 DEBUG:boto:Token: None 2024-03-04T16:38:23.508 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.508 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d74c4e8c48440771-0065e5f8ff-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.510 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.510 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.510 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.510 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.510 DEBUG:boto:Data: 2024-03-04T16:38:23.511 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.511 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.511 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.511 DEBUG:boto:Params: {} 2024-03-04T16:38:23.511 DEBUG:boto:Token: None 2024-03-04T16:38:23.511 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5edd039c84fad29-0065e5f8ff-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.513 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:23.514 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.514 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.514 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.514 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.514 DEBUG:boto:Method: GET 2024-03-04T16:38:23.514 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.514 DEBUG:boto:Data: 2024-03-04T16:38:23.514 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.514 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.514 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.514 DEBUG:boto:Params: {} 2024-03-04T16:38:23.514 DEBUG:boto:Token: None 2024-03-04T16:38:23.515 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.515 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c84def4bc8030b9-0065e5f8ff-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.517 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:23.518 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.518 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.518 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.518 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.518 DEBUG:boto:Method: GET 2024-03-04T16:38:23.518 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.518 DEBUG:boto:Data: 2024-03-04T16:38:23.518 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.518 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.518 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.519 DEBUG:boto:Params: {} 2024-03-04T16:38:23.519 DEBUG:boto:Token: None 2024-03-04T16:38:23.519 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a10444c7ba898a6-0065e5f8ff-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.521 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.521 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.521 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.522 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.522 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.522 DEBUG:boto:Method: GET 2024-03-04T16:38:23.522 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.522 DEBUG:boto:Data: 2024-03-04T16:38:23.522 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.522 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.522 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.522 DEBUG:boto:Params: {} 2024-03-04T16:38:23.522 DEBUG:boto:Token: None 2024-03-04T16:38:23.523 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.523 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a586f349e790377-0065e5f8ff-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.525 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.525 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.525 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.525 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.525 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.526 DEBUG:boto:Method: GET 2024-03-04T16:38:23.526 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.526 DEBUG:boto:Data: 2024-03-04T16:38:23.526 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.526 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.526 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.526 DEBUG:boto:Params: {} 2024-03-04T16:38:23.526 DEBUG:boto:Token: None 2024-03-04T16:38:23.526 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.526 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8e865ba3f5001f-0065e5f8ff-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.572 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:23.573 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.573 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.573 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.573 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.573 DEBUG:boto:Data: 2024-03-04T16:38:23.573 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.573 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.574 DEBUG:boto:Params: {} 2024-03-04T16:38:23.574 DEBUG:boto:Token: None 2024-03-04T16:38:23.574 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.574 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.576 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000eb20a5a8ac38df85-0065e5f8ff-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.576 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.576 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.576 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.576 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.576 DEBUG:boto:Data: 2024-03-04T16:38:23.577 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:38:23.577 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.577 DEBUG:boto:Params: {} 2024-03-04T16:38:23.577 DEBUG:boto:Token: None 2024-03-04T16:38:23.577 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.577 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.578 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000080b4d3413c61a359-0065e5f8ff-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-28 zones={test-zone3, test-zone1} 2024-03-04T16:38:23.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-28 zones={test-zone3, test-zone2} 2024-03-04T16:38:23.579 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.579 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.579 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.579 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.579 DEBUG:boto:Data: 2024-03-04T16:38:23.579 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.579 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.580 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.580 DEBUG:boto:Params: {} 2024-03-04T16:38:23.580 DEBUG:boto:Token: None 2024-03-04T16:38:23.580 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.580 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c306c496d69484f-0065e5f8ff-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.582 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.583 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.583 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.583 DEBUG:boto:Path: /sbnvju-28/ 2024-03-04T16:38:23.583 DEBUG:boto:Data: 2024-03-04T16:38:23.583 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.583 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.583 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.583 DEBUG:boto:Params: {} 2024-03-04T16:38:23.583 DEBUG:boto:Token: None 2024-03-04T16:38:23.583 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/ 2024-03-04T16:38:23.583 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU= 2024-03-04T16:38:23.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OWIZ1uoet3sr9JOYbj/8la4PqZU=', 'Content-Length': '0'} 2024-03-04T16:38:23.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e303975675923ac-0065e5f8ff-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.585 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:38:23.586 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.586 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.586 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.586 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.586 DEBUG:boto:Method: GET 2024-03-04T16:38:23.586 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.586 DEBUG:boto:Data: 2024-03-04T16:38:23.586 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.586 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.586 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.586 DEBUG:boto:Params: {} 2024-03-04T16:38:23.587 DEBUG:boto:Token: None 2024-03-04T16:38:23.587 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.587 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c1a1ea7dd697fc-0065e5f8ff-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.632 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.632 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:38:23.633 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.633 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.633 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.633 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.633 DEBUG:boto:Method: GET 2024-03-04T16:38:23.633 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.633 DEBUG:boto:Data: 2024-03-04T16:38:23.633 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.633 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.633 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.633 DEBUG:boto:Params: {} 2024-03-04T16:38:23.634 DEBUG:boto:Token: None 2024-03-04T16:38:23.634 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.634 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f53abdeef23001c3-0065e5f8ff-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.636 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:38:23.637 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.637 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.637 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.637 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.637 DEBUG:boto:Method: GET 2024-03-04T16:38:23.637 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.637 DEBUG:boto:Data: 2024-03-04T16:38:23.637 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.637 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.637 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.637 DEBUG:boto:Params: {} 2024-03-04T16:38:23.638 DEBUG:boto:Token: None 2024-03-04T16:38:23.638 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.638 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840f60e80b0b0e10-0065e5f8ff-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.640 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.641 DEBUG:boto:path=/sbnvju-28/ 2024-03-04T16:38:23.641 DEBUG:boto:auth_path=/sbnvju-28/ 2024-03-04T16:38:23.641 DEBUG:boto:path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.641 DEBUG:boto:auth_path=/sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.641 DEBUG:boto:Method: GET 2024-03-04T16:38:23.641 DEBUG:boto:Path: /sbnvju-28/?versions&max-keys=999 2024-03-04T16:38:23.641 DEBUG:boto:Data: 2024-03-04T16:38:23.641 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.641 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.641 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.642 DEBUG:boto:Params: {} 2024-03-04T16:38:23.642 DEBUG:boto:Token: None 2024-03-04T16:38:23.642 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/?versions 2024-03-04T16:38:23.642 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE= 2024-03-04T16:38:23.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xNE5MVshvWCjA1YEqX8b1VfPOEE=', 'Content-Length': '0'} 2024-03-04T16:38:23.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041997649fde55946-0065e5f8ff-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.688 DEBUG:boto:b'sbnvju-28999falseobjnulltrue2024-03-04T16:37:46.568Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:38:23.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:38:23.689 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.689 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.689 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.689 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.689 DEBUG:boto:Data: 2024-03-04T16:38:23.689 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.689 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:38:23.689 DEBUG:boto:Port: 8001 2024-03-04T16:38:23.689 DEBUG:boto:Params: {} 2024-03-04T16:38:23.689 DEBUG:boto:Token: None 2024-03-04T16:38:23.689 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.690 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.691 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006ba62db0a76c3d2e-0065e5f8ff-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.692 DEBUG:boto:path=/sbnvju-28/obj 2024-03-04T16:38:23.692 DEBUG:boto:auth_path=/sbnvju-28/obj 2024-03-04T16:38:23.692 DEBUG:boto:Method: HEAD 2024-03-04T16:38:23.692 DEBUG:boto:Path: /sbnvju-28/obj 2024-03-04T16:38:23.692 DEBUG:boto:Data: 2024-03-04T16:38:23.692 DEBUG:boto:Headers: {} 2024-03-04T16:38:23.692 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:38:23.692 DEBUG:boto:Port: 8000 2024-03-04T16:38:23.692 DEBUG:boto:Params: {} 2024-03-04T16:38:23.692 DEBUG:boto:Token: None 2024-03-04T16:38:23.692 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:38:23 GMT /sbnvju-28/obj 2024-03-04T16:38:23.693 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso= 2024-03-04T16:38:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:38:23 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ssHIeIJj2PTzNwGaCZdKTxCMUso=', 'Content-Length': '0'} 2024-03-04T16:38:23.694 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000061775caec0efc228-0065e5f8ff-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:38:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:38:23.694 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-28 zones={test-zone3, test-zone2} 2024-03-04T16:38:23.694 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=sbnvju-29 2024-03-04T16:38:23.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:23.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:23.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:23.810+0000 7fdecb62e800 1 bucket sync caught up with source: 2024-03-04T16:38:23.797 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000001.29.14, , ] 2024-03-04T16:38:23.797 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000001.29.14, , ] 2024-03-04T16:38:23.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:38:23.810+0000 7fdecb62e800 0 bucket checkpoint complete 2024-03-04T16:38:23.817 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=sbnvju-29 2024-03-04T16:38:23.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:38:23.818 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:38:23.874 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:38:23.874 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:38:23.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:23.936+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:28.937+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:33.937+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:38.938+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:43.939+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:48.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:48.940+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:53.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:53.940+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:38:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:38:58.941+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:03.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:03.942+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:08.943+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:13.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:13.943+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:18.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:18.944+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:23.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:23.945+0000 7fc43f530800 1 waiting to reach incremental sync.. 2024-03-04T16:39:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:28.947+0000 7fc43f530800 1 bucket sync caught up with source: 2024-03-04T16:39:28.933 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.29.14, , ] 2024-03-04T16:39:28.933 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.29.14, , ] 2024-03-04T16:39:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:28.947+0000 7fc43f530800 0 bucket checkpoint complete 2024-03-04T16:39:28.958 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=sbnvju-29 2024-03-04T16:39:28.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:28.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:28.959 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:39:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.089+0000 7fc55d109800 1 bucket sync caught up with source: 2024-03-04T16:39:29.076 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.26.14, , ] 2024-03-04T16:39:29.076 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.26.14, , ] 2024-03-04T16:39:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.089+0000 7fc55d109800 0 bucket checkpoint complete 2024-03-04T16:39:29.096 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=sbnvju-29 2024-03-04T16:39:29.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.096 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:39:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:29.151 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.216+0000 7faea1722800 1 bucket sync caught up with source: 2024-03-04T16:39:29.202 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000001.26.14, , ] 2024-03-04T16:39:29.202 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000001.26.14, , ] 2024-03-04T16:39:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.216+0000 7faea1722800 0 bucket checkpoint complete 2024-03-04T16:39:29.223 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=sbnvju-29 2024-03-04T16:39:29.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.224 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-03-04T16:39:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.345+0000 7f6b3228c800 1 bucket sync caught up with source: 2024-03-04T16:39:29.332 INFO:teuthology.orchestra.run.smithi027.stderr: local status: [, , , , , , , , 00000000003.82.7, , ] 2024-03-04T16:39:29.332 INFO:teuthology.orchestra.run.smithi027.stderr: remote markers: [, , , , , , , , 00000000003.82.7, , ] 2024-03-04T16:39:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T16:39:29.345+0000 7f6b3228c800 0 bucket checkpoint complete 2024-03-04T16:39:29.351 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=sbnvju-29 2024-03-04T16:39:29.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T16:39:29.351 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T16:39:29.411 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:29.411 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:29.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:29.478+0000 7f628f330800 1 bucket sync caught up with source: 2024-03-04T16:39:29.464 INFO:teuthology.orchestra.run.smithi071.stderr: local status: [, , , , , , , , 00000000003.82.7, , ] 2024-03-04T16:39:29.464 INFO:teuthology.orchestra.run.smithi071.stderr: remote markers: [, , , , , , , , 00000000003.82.7, , ] 2024-03-04T16:39:29.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:29.478+0000 7f628f330800 0 bucket checkpoint complete 2024-03-04T16:39:29.483 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-29 zones={test-zone2, test-zone1} 2024-03-04T16:39:29.483 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.484 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.484 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.484 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.484 DEBUG:boto:Data: 2024-03-04T16:39:29.484 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.484 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.484 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.484 DEBUG:boto:Params: {} 2024-03-04T16:39:29.485 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:39:29.485 DEBUG:boto:Token: None 2024-03-04T16:39:29.485 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.486 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000172eb7199b33534c-0065e5f941-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.491 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.491 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.491 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.491 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.491 DEBUG:boto:Data: 2024-03-04T16:39:29.491 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.491 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.491 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.491 DEBUG:boto:Params: {} 2024-03-04T16:39:29.492 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:39:29.492 DEBUG:boto:Token: None 2024-03-04T16:39:29.492 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.492 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0fea3fd7788e284-0065e5f941-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:39:29.496 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.496 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.496 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.496 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.496 DEBUG:boto:Method: GET 2024-03-04T16:39:29.496 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.496 DEBUG:boto:Data: 2024-03-04T16:39:29.496 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.496 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.497 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.497 DEBUG:boto:Params: {} 2024-03-04T16:39:29.497 DEBUG:boto:Token: None 2024-03-04T16:39:29.497 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.497 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc27cd344185fcdd-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.499 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:39:29.501 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.501 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.501 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.501 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.501 DEBUG:boto:Method: GET 2024-03-04T16:39:29.501 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.501 DEBUG:boto:Data: 2024-03-04T16:39:29.501 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.501 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.501 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.501 DEBUG:boto:Params: {} 2024-03-04T16:39:29.502 DEBUG:boto:Token: None 2024-03-04T16:39:29.502 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.502 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001b28d5f5bf4bdd9-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.504 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.504 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.505 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.505 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.505 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.505 DEBUG:boto:Method: GET 2024-03-04T16:39:29.505 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.505 DEBUG:boto:Data: 2024-03-04T16:39:29.505 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.505 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.505 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.505 DEBUG:boto:Params: {} 2024-03-04T16:39:29.505 DEBUG:boto:Token: None 2024-03-04T16:39:29.506 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.506 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea874f4888b2c728-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.508 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.508 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.508 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.508 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.509 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.509 DEBUG:boto:Method: GET 2024-03-04T16:39:29.509 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.509 DEBUG:boto:Data: 2024-03-04T16:39:29.509 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.509 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.509 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.509 DEBUG:boto:Params: {} 2024-03-04T16:39:29.509 DEBUG:boto:Token: None 2024-03-04T16:39:29.509 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.509 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834d8863213667d7-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.512 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:39:29.512 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.512 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.512 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.512 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.513 DEBUG:boto:Data: 2024-03-04T16:39:29.513 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.513 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.513 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.513 DEBUG:boto:Params: {} 2024-03-04T16:39:29.513 DEBUG:boto:Token: None 2024-03-04T16:39:29.513 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.513 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.514 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000078ccf024ca221dd9-0065e5f941-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.515 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.515 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.515 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.515 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.515 DEBUG:boto:Data: 2024-03-04T16:39:29.515 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.515 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.515 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.515 DEBUG:boto:Params: {} 2024-03-04T16:39:29.516 DEBUG:boto:Token: None 2024-03-04T16:39:29.516 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.516 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.517 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b00f474eb5ea89f8-0065e5f941-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.517 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-29 zones={test-zone2, test-zone1} 2024-03-04T16:39:29.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-29 zones={test-zone3, test-zone1} 2024-03-04T16:39:29.518 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.518 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.518 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.518 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.518 DEBUG:boto:Data: 2024-03-04T16:39:29.518 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.518 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.518 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.518 DEBUG:boto:Params: {} 2024-03-04T16:39:29.518 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:39:29.519 DEBUG:boto:Token: None 2024-03-04T16:39:29.519 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.519 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b72872cb24721a63-0065e5f941-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.523 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.523 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.523 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.523 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.523 DEBUG:boto:Data: 2024-03-04T16:39:29.523 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.523 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.523 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.523 DEBUG:boto:Params: {} 2024-03-04T16:39:29.523 DEBUG:boto:Token: None 2024-03-04T16:39:29.524 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.524 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce26b7c7673f9a1d-0065e5f941-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.526 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:39:29.526 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.526 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.526 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.526 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.526 DEBUG:boto:Method: GET 2024-03-04T16:39:29.526 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.527 DEBUG:boto:Data: 2024-03-04T16:39:29.527 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.527 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.527 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.527 DEBUG:boto:Params: {} 2024-03-04T16:39:29.527 DEBUG:boto:Token: None 2024-03-04T16:39:29.527 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.527 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059cede7b51a6e4ea-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.530 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:39:29.530 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.530 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.531 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.531 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.531 DEBUG:boto:Method: GET 2024-03-04T16:39:29.531 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.531 DEBUG:boto:Data: 2024-03-04T16:39:29.531 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.531 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.531 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.531 DEBUG:boto:Params: {} 2024-03-04T16:39:29.531 DEBUG:boto:Token: None 2024-03-04T16:39:29.531 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.532 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020ee70f908bcb80d-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.534 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.534 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.534 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.534 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.534 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.534 DEBUG:boto:Method: GET 2024-03-04T16:39:29.535 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.535 DEBUG:boto:Data: 2024-03-04T16:39:29.535 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.535 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.535 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.535 DEBUG:boto:Params: {} 2024-03-04T16:39:29.535 DEBUG:boto:Token: None 2024-03-04T16:39:29.535 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.535 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049364e67f3d333c1-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.537 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.538 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.538 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.538 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.538 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.538 DEBUG:boto:Method: GET 2024-03-04T16:39:29.538 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.538 DEBUG:boto:Data: 2024-03-04T16:39:29.538 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.539 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.539 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.539 DEBUG:boto:Params: {} 2024-03-04T16:39:29.539 DEBUG:boto:Token: None 2024-03-04T16:39:29.539 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.539 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bebfc96bc7854e11-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.542 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:39:29.542 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.542 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.542 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.542 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.542 DEBUG:boto:Data: 2024-03-04T16:39:29.542 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.543 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.543 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.543 DEBUG:boto:Params: {} 2024-03-04T16:39:29.543 DEBUG:boto:Token: None 2024-03-04T16:39:29.543 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.543 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.544 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a3f204763c406176-0065e5f941-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.545 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.545 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.545 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.545 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.545 DEBUG:boto:Data: 2024-03-04T16:39:29.545 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.545 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.545 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.545 DEBUG:boto:Params: {} 2024-03-04T16:39:29.545 DEBUG:boto:Token: None 2024-03-04T16:39:29.546 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.546 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.547 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ebd9f610b5ef5653-0065e5f941-4341-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-29 zones={test-zone3, test-zone1} 2024-03-04T16:39:29.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=sbnvju-29 zones={test-zone3, test-zone2} 2024-03-04T16:39:29.547 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.548 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.548 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.548 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.548 DEBUG:boto:Data: 2024-03-04T16:39:29.548 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.548 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.548 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.548 DEBUG:boto:Params: {} 2024-03-04T16:39:29.548 DEBUG:boto:Token: None 2024-03-04T16:39:29.548 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.549 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000799a2da70bef6c3f-0065e5f941-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.551 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.551 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.551 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.551 DEBUG:boto:Path: /sbnvju-29/ 2024-03-04T16:39:29.551 DEBUG:boto:Data: 2024-03-04T16:39:29.551 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.551 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.551 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.551 DEBUG:boto:Params: {} 2024-03-04T16:39:29.551 DEBUG:boto:Token: None 2024-03-04T16:39:29.552 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/ 2024-03-04T16:39:29.552 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I= 2024-03-04T16:39:29.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2SqE36luXmgVU2G/kDeaSlX5Y0I=', 'Content-Length': '0'} 2024-03-04T16:39:29.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6d26a060517870b-0065e5f941-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-03-04T16:39:29.554 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.554 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.554 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.554 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.554 DEBUG:boto:Method: GET 2024-03-04T16:39:29.555 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.555 DEBUG:boto:Data: 2024-03-04T16:39:29.555 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.555 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.555 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.555 DEBUG:boto:Params: {} 2024-03-04T16:39:29.555 DEBUG:boto:Token: None 2024-03-04T16:39:29.555 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.555 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086405fb9bf9b774-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.557 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-03-04T16:39:29.558 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.558 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.558 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.558 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.558 DEBUG:boto:Method: GET 2024-03-04T16:39:29.558 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.559 DEBUG:boto:Data: 2024-03-04T16:39:29.559 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.559 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.559 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.559 DEBUG:boto:Params: {} 2024-03-04T16:39:29.559 DEBUG:boto:Token: None 2024-03-04T16:39:29.559 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.559 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8dc2ebe1bc7d80-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.562 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-03-04T16:39:29.562 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.562 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.562 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.562 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.562 DEBUG:boto:Method: GET 2024-03-04T16:39:29.563 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.563 DEBUG:boto:Data: 2024-03-04T16:39:29.563 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.563 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.563 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.563 DEBUG:boto:Params: {} 2024-03-04T16:39:29.563 DEBUG:boto:Token: None 2024-03-04T16:39:29.563 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.563 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2fc6f70ca0282fb-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.566 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.566 DEBUG:boto:path=/sbnvju-29/ 2024-03-04T16:39:29.566 DEBUG:boto:auth_path=/sbnvju-29/ 2024-03-04T16:39:29.566 DEBUG:boto:path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.566 DEBUG:boto:auth_path=/sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.566 DEBUG:boto:Method: GET 2024-03-04T16:39:29.566 DEBUG:boto:Path: /sbnvju-29/?versions&max-keys=999 2024-03-04T16:39:29.566 DEBUG:boto:Data: 2024-03-04T16:39:29.566 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.566 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.567 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.567 DEBUG:boto:Params: {} 2024-03-04T16:39:29.567 DEBUG:boto:Token: None 2024-03-04T16:39:29.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/?versions 2024-03-04T16:39:29.567 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4= 2024-03-04T16:39:29.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:asnbS2qzg2+PfYoYb+W56dsV5R4=', 'Content-Length': '0'} 2024-03-04T16:39:29.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006baf6fe2e6f93751-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.569 DEBUG:boto:b'sbnvju-29999falseobjnulltrue2024-03-04T16:37:46.591Zrgw-multisite-test-userMultisite Test UserNormal' 2024-03-04T16:39:29.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-03-04T16:39:29.570 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.570 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.570 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.570 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.570 DEBUG:boto:Data: 2024-03-04T16:39:29.570 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.570 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.570 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.570 DEBUG:boto:Params: {} 2024-03-04T16:39:29.570 DEBUG:boto:Token: None 2024-03-04T16:39:29.570 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.571 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.572 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000033189b295c83ce34-0065e5f941-4348-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.572 DEBUG:boto:path=/sbnvju-29/obj 2024-03-04T16:39:29.572 DEBUG:boto:auth_path=/sbnvju-29/obj 2024-03-04T16:39:29.572 DEBUG:boto:Method: HEAD 2024-03-04T16:39:29.572 DEBUG:boto:Path: /sbnvju-29/obj 2024-03-04T16:39:29.572 DEBUG:boto:Data: 2024-03-04T16:39:29.572 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.572 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.573 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.573 DEBUG:boto:Params: {} 2024-03-04T16:39:29.573 DEBUG:boto:Token: None 2024-03-04T16:39:29.573 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-29/obj 2024-03-04T16:39:29.573 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI= 2024-03-04T16:39:29.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5QeMbdJKfQJeYlxIYFbCJiacvjI=', 'Content-Length': '0'} 2024-03-04T16:39:29.575 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fd9ab78bfaa988bc-0065e5f941-4307-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.575 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=sbnvju-29 zones={test-zone3, test-zone2} 2024-03-04T16:39:29.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-03-04T16:39:29.577 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-30 2024-03-04T16:39:29.577 DEBUG:boto:path=/sbnvju-30/ 2024-03-04T16:39:29.577 DEBUG:boto:auth_path=/sbnvju-30/ 2024-03-04T16:39:29.577 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.577 DEBUG:boto:Path: /sbnvju-30/ 2024-03-04T16:39:29.577 DEBUG:boto:Data: 2024-03-04T16:39:29.577 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.577 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.577 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.578 DEBUG:boto:Params: {} 2024-03-04T16:39:29.578 DEBUG:boto:Token: None 2024-03-04T16:39:29.578 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-30/ 2024-03-04T16:39:29.578 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:35CBWm4Ocymo5eKymSspG3qhapU= 2024-03-04T16:39:29.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:35CBWm4Ocymo5eKymSspG3qhapU=', 'Content-Length': '0'} 2024-03-04T16:39:29.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8cbe1115f1e6203-0065e5f941-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.592 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-31 2024-03-04T16:39:29.592 DEBUG:boto:path=/sbnvju-31/ 2024-03-04T16:39:29.592 DEBUG:boto:auth_path=/sbnvju-31/ 2024-03-04T16:39:29.592 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.592 DEBUG:boto:Path: /sbnvju-31/ 2024-03-04T16:39:29.593 DEBUG:boto:Data: 2024-03-04T16:39:29.593 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.593 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.593 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.593 DEBUG:boto:Params: {} 2024-03-04T16:39:29.593 DEBUG:boto:Token: None 2024-03-04T16:39:29.594 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-31/ 2024-03-04T16:39:29.594 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5FnUfhHOy5dZ4yEHJChe5jOpc0Q= 2024-03-04T16:39:29.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5FnUfhHOy5dZ4yEHJChe5jOpc0Q=', 'Content-Length': '0'} 2024-03-04T16:39:29.617 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004be7766662800195-0065e5f941-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.617 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-32 2024-03-04T16:39:29.617 DEBUG:boto:path=/sbnvju-32/ 2024-03-04T16:39:29.617 DEBUG:boto:auth_path=/sbnvju-32/ 2024-03-04T16:39:29.618 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.618 DEBUG:boto:Path: /sbnvju-32/ 2024-03-04T16:39:29.618 DEBUG:boto:Data: 2024-03-04T16:39:29.618 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.618 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.618 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.618 DEBUG:boto:Params: {} 2024-03-04T16:39:29.618 DEBUG:boto:Token: None 2024-03-04T16:39:29.618 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-32/ 2024-03-04T16:39:29.619 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ltvpq/43Y6KgKy9RO/llG1Qp96s= 2024-03-04T16:39:29.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ltvpq/43Y6KgKy9RO/llG1Qp96s=', 'Content-Length': '0'} 2024-03-04T16:39:29.641 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be6cc0a1b2056558-0065e5f941-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.641 DEBUG:boto:path=/sbnvju-30/ 2024-03-04T16:39:29.641 DEBUG:boto:auth_path=/sbnvju-30/ 2024-03-04T16:39:29.641 DEBUG:boto:path=/sbnvju-30/?versioning 2024-03-04T16:39:29.641 DEBUG:boto:auth_path=/sbnvju-30/?versioning 2024-03-04T16:39:29.642 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.642 DEBUG:boto:Path: /sbnvju-30/?versioning 2024-03-04T16:39:29.642 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:39:29.642 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.642 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.642 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.642 DEBUG:boto:Params: {} 2024-03-04T16:39:29.642 DEBUG:boto:Token: None 2024-03-04T16:39:29.642 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-30/?versioning 2024-03-04T16:39:29.643 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5x2QpcbxLqjK81Akqe8iuRcDr4I= 2024-03-04T16:39:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5x2QpcbxLqjK81Akqe8iuRcDr4I=', 'Content-Length': '228'} 2024-03-04T16:39:29.648 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d5fabfa12a9561a-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.648 DEBUG:boto:path=/sbnvju-30/ 2024-03-04T16:39:29.648 DEBUG:boto:auth_path=/sbnvju-30/ 2024-03-04T16:39:29.648 DEBUG:boto:path=/sbnvju-30/?versioning 2024-03-04T16:39:29.648 DEBUG:boto:auth_path=/sbnvju-30/?versioning 2024-03-04T16:39:29.648 DEBUG:boto:Method: GET 2024-03-04T16:39:29.648 DEBUG:boto:Path: /sbnvju-30/?versioning 2024-03-04T16:39:29.648 DEBUG:boto:Data: 2024-03-04T16:39:29.649 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.649 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.649 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.649 DEBUG:boto:Params: {} 2024-03-04T16:39:29.649 DEBUG:boto:Token: None 2024-03-04T16:39:29.649 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-30/?versioning 2024-03-04T16:39:29.649 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:F3ZElg8XZEdOIYCFdct5NhlU9VI= 2024-03-04T16:39:29.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F3ZElg8XZEdOIYCFdct5NhlU9VI=', 'Content-Length': '0'} 2024-03-04T16:39:29.650 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d8e2de84c98a544-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.692 DEBUG:boto:EnabledDisabled 2024-03-04T16:39:29.693 DEBUG:boto:path=/sbnvju-31/ 2024-03-04T16:39:29.693 DEBUG:boto:auth_path=/sbnvju-31/ 2024-03-04T16:39:29.693 DEBUG:boto:path=/sbnvju-31/?versioning 2024-03-04T16:39:29.693 DEBUG:boto:auth_path=/sbnvju-31/?versioning 2024-03-04T16:39:29.693 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.693 DEBUG:boto:Path: /sbnvju-31/?versioning 2024-03-04T16:39:29.693 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:39:29.693 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.693 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.694 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.694 DEBUG:boto:Params: {} 2024-03-04T16:39:29.694 DEBUG:boto:Token: None 2024-03-04T16:39:29.694 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-31/?versioning 2024-03-04T16:39:29.694 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:U83f/xlfkJzFGfjWdvJWMP0ejtk= 2024-03-04T16:39:29.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:U83f/xlfkJzFGfjWdvJWMP0ejtk=', 'Content-Length': '228'} 2024-03-04T16:39:29.714 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8e6062f47ab3524-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.714 DEBUG:boto:path=/sbnvju-31/ 2024-03-04T16:39:29.714 DEBUG:boto:auth_path=/sbnvju-31/ 2024-03-04T16:39:29.715 DEBUG:boto:path=/sbnvju-31/?versioning 2024-03-04T16:39:29.715 DEBUG:boto:auth_path=/sbnvju-31/?versioning 2024-03-04T16:39:29.715 DEBUG:boto:Method: GET 2024-03-04T16:39:29.715 DEBUG:boto:Path: /sbnvju-31/?versioning 2024-03-04T16:39:29.715 DEBUG:boto:Data: 2024-03-04T16:39:29.715 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.715 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.715 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.715 DEBUG:boto:Params: {} 2024-03-04T16:39:29.715 DEBUG:boto:Token: None 2024-03-04T16:39:29.715 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-31/?versioning 2024-03-04T16:39:29.716 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:zHjgAlUwSCcUdvFKTknCipFfSl4= 2024-03-04T16:39:29.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zHjgAlUwSCcUdvFKTknCipFfSl4=', 'Content-Length': '0'} 2024-03-04T16:39:29.717 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ee53be8bbeba3d6b-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.717 DEBUG:boto:EnabledDisabled 2024-03-04T16:39:29.717 DEBUG:boto:path=/sbnvju-32/ 2024-03-04T16:39:29.717 DEBUG:boto:auth_path=/sbnvju-32/ 2024-03-04T16:39:29.718 DEBUG:boto:path=/sbnvju-32/?versioning 2024-03-04T16:39:29.718 DEBUG:boto:auth_path=/sbnvju-32/?versioning 2024-03-04T16:39:29.718 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.718 DEBUG:boto:Path: /sbnvju-32/?versioning 2024-03-04T16:39:29.718 DEBUG:boto:Data: Enabled Disabled 2024-03-04T16:39:29.718 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.718 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.718 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.718 DEBUG:boto:Params: {} 2024-03-04T16:39:29.718 DEBUG:boto:Token: None 2024-03-04T16:39:29.718 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-32/?versioning 2024-03-04T16:39:29.719 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QMcEKbN4Q/klBw3Drpuwh+zrS48= 2024-03-04T16:39:29.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QMcEKbN4Q/klBw3Drpuwh+zrS48=', 'Content-Length': '228'} 2024-03-04T16:39:29.726 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051d9fcffb57b3ae8-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.726 DEBUG:boto:path=/sbnvju-32/ 2024-03-04T16:39:29.726 DEBUG:boto:auth_path=/sbnvju-32/ 2024-03-04T16:39:29.726 DEBUG:boto:path=/sbnvju-32/?versioning 2024-03-04T16:39:29.726 DEBUG:boto:auth_path=/sbnvju-32/?versioning 2024-03-04T16:39:29.726 DEBUG:boto:Method: GET 2024-03-04T16:39:29.726 DEBUG:boto:Path: /sbnvju-32/?versioning 2024-03-04T16:39:29.727 DEBUG:boto:Data: 2024-03-04T16:39:29.727 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.727 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.727 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.727 DEBUG:boto:Params: {} 2024-03-04T16:39:29.727 DEBUG:boto:Token: None 2024-03-04T16:39:29.727 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-32/?versioning 2024-03-04T16:39:29.727 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:i8MaSUnpzwlmEWcjGAQGg0QrTtM= 2024-03-04T16:39:29.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:i8MaSUnpzwlmEWcjGAQGg0QrTtM=', 'Content-Length': '0'} 2024-03-04T16:39:29.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e01c97fcea0ddc80-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.772 DEBUG:boto:EnabledDisabled 2024-03-04T16:39:29.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-03-04T16:39:29.773 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-33 2024-03-04T16:39:29.773 DEBUG:boto:path=/sbnvju-33/ 2024-03-04T16:39:29.773 DEBUG:boto:auth_path=/sbnvju-33/ 2024-03-04T16:39:29.773 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.773 DEBUG:boto:Path: /sbnvju-33/ 2024-03-04T16:39:29.773 DEBUG:boto:Data: 2024-03-04T16:39:29.773 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.773 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.774 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.774 DEBUG:boto:Params: {} 2024-03-04T16:39:29.774 DEBUG:boto:Token: None 2024-03-04T16:39:29.774 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-33/ 2024-03-04T16:39:29.774 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ZaZDhujwSq0xjtNf4eNNvTU3nXw= 2024-03-04T16:39:29.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZaZDhujwSq0xjtNf4eNNvTU3nXw=', 'Content-Length': '0'} 2024-03-04T16:39:29.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b2a093c5ce30bb0-0065e5f941-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.788 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-34 2024-03-04T16:39:29.788 DEBUG:boto:path=/sbnvju-34/ 2024-03-04T16:39:29.788 DEBUG:boto:auth_path=/sbnvju-34/ 2024-03-04T16:39:29.788 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.788 DEBUG:boto:Path: /sbnvju-34/ 2024-03-04T16:39:29.789 DEBUG:boto:Data: 2024-03-04T16:39:29.789 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.789 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.789 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.789 DEBUG:boto:Params: {} 2024-03-04T16:39:29.789 DEBUG:boto:Token: None 2024-03-04T16:39:29.789 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-34/ 2024-03-04T16:39:29.789 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sGtukOyxHlJ7hh2KeGB1yzhS8FU= 2024-03-04T16:39:29.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sGtukOyxHlJ7hh2KeGB1yzhS8FU=', 'Content-Length': '0'} 2024-03-04T16:39:29.814 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3afec53f6cc11c8-0065e5f941-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.814 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-35 2024-03-04T16:39:29.814 DEBUG:boto:path=/sbnvju-35/ 2024-03-04T16:39:29.814 DEBUG:boto:auth_path=/sbnvju-35/ 2024-03-04T16:39:29.814 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.814 DEBUG:boto:Path: /sbnvju-35/ 2024-03-04T16:39:29.814 DEBUG:boto:Data: 2024-03-04T16:39:29.814 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.815 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.815 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.815 DEBUG:boto:Params: {} 2024-03-04T16:39:29.815 DEBUG:boto:Token: None 2024-03-04T16:39:29.815 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-35/ 2024-03-04T16:39:29.815 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:lHIx895Vsg+6saqMK9FuLvsMiZQ= 2024-03-04T16:39:29.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lHIx895Vsg+6saqMK9FuLvsMiZQ=', 'Content-Length': '0'} 2024-03-04T16:39:29.837 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c1c1b6bc229f8c4-0065e5f941-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.837 DEBUG:boto:path=/sbnvju-33/ 2024-03-04T16:39:29.838 DEBUG:boto:auth_path=/sbnvju-33/ 2024-03-04T16:39:29.838 DEBUG:boto:path=/sbnvju-33/?acl 2024-03-04T16:39:29.838 DEBUG:boto:auth_path=/sbnvju-33/?acl 2024-03-04T16:39:29.838 DEBUG:boto:Method: GET 2024-03-04T16:39:29.838 DEBUG:boto:Path: /sbnvju-33/?acl 2024-03-04T16:39:29.838 DEBUG:boto:Data: 2024-03-04T16:39:29.838 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.838 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.838 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.838 DEBUG:boto:Params: {} 2024-03-04T16:39:29.838 DEBUG:boto:Token: None 2024-03-04T16:39:29.839 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-33/?acl 2024-03-04T16:39:29.839 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aewejjjnniDXCab7IzJnE491rSk= 2024-03-04T16:39:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aewejjjnniDXCab7IzJnE491rSk=', 'Content-Length': '0'} 2024-03-04T16:39:29.840 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7d914da9a6f969c-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.840 DEBUG:boto:path=/sbnvju-33/ 2024-03-04T16:39:29.840 DEBUG:boto:auth_path=/sbnvju-33/ 2024-03-04T16:39:29.841 DEBUG:boto:path=/sbnvju-33/?acl 2024-03-04T16:39:29.841 DEBUG:boto:auth_path=/sbnvju-33/?acl 2024-03-04T16:39:29.841 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.841 DEBUG:boto:Path: /sbnvju-33/?acl 2024-03-04T16:39:29.841 DEBUG:boto:Data: 2024-03-04T16:39:29.841 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:39:29.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.841 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.841 DEBUG:boto:Params: {} 2024-03-04T16:39:29.841 DEBUG:boto:Token: None 2024-03-04T16:39:29.841 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT x-amz-acl:public-read /sbnvju-33/?acl 2024-03-04T16:39:29.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:f4shDJBLs7aZLzHASUXUdzMS4TM= 2024-03-04T16:39:29.842 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:f4shDJBLs7aZLzHASUXUdzMS4TM=', 'Content-Length': '0'} 2024-03-04T16:39:29.846 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000017e03152a00bd37c-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.846 DEBUG:boto:path=/sbnvju-33/ 2024-03-04T16:39:29.846 DEBUG:boto:auth_path=/sbnvju-33/ 2024-03-04T16:39:29.847 DEBUG:boto:path=/sbnvju-33/?acl 2024-03-04T16:39:29.847 DEBUG:boto:auth_path=/sbnvju-33/?acl 2024-03-04T16:39:29.847 DEBUG:boto:Method: GET 2024-03-04T16:39:29.847 DEBUG:boto:Path: /sbnvju-33/?acl 2024-03-04T16:39:29.847 DEBUG:boto:Data: 2024-03-04T16:39:29.847 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.847 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.847 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.847 DEBUG:boto:Params: {} 2024-03-04T16:39:29.847 DEBUG:boto:Token: None 2024-03-04T16:39:29.847 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-33/?acl 2024-03-04T16:39:29.848 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aewejjjnniDXCab7IzJnE491rSk= 2024-03-04T16:39:29.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aewejjjnniDXCab7IzJnE491rSk=', 'Content-Length': '0'} 2024-03-04T16:39:29.849 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f35c12a076504a3-0065e5f941-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.849 DEBUG:boto:path=/sbnvju-34/ 2024-03-04T16:39:29.849 DEBUG:boto:auth_path=/sbnvju-34/ 2024-03-04T16:39:29.849 DEBUG:boto:path=/sbnvju-34/?acl 2024-03-04T16:39:29.849 DEBUG:boto:auth_path=/sbnvju-34/?acl 2024-03-04T16:39:29.849 DEBUG:boto:Method: GET 2024-03-04T16:39:29.850 DEBUG:boto:Path: /sbnvju-34/?acl 2024-03-04T16:39:29.850 DEBUG:boto:Data: 2024-03-04T16:39:29.850 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.850 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.850 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.850 DEBUG:boto:Params: {} 2024-03-04T16:39:29.850 DEBUG:boto:Token: None 2024-03-04T16:39:29.850 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-34/?acl 2024-03-04T16:39:29.850 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dEBKiEKiZgeiCPKHmt9copsqir8= 2024-03-04T16:39:29.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dEBKiEKiZgeiCPKHmt9copsqir8=', 'Content-Length': '0'} 2024-03-04T16:39:29.852 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e196fa3777b2326d-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.852 DEBUG:boto:path=/sbnvju-34/ 2024-03-04T16:39:29.852 DEBUG:boto:auth_path=/sbnvju-34/ 2024-03-04T16:39:29.852 DEBUG:boto:path=/sbnvju-34/?acl 2024-03-04T16:39:29.852 DEBUG:boto:auth_path=/sbnvju-34/?acl 2024-03-04T16:39:29.852 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.852 DEBUG:boto:Path: /sbnvju-34/?acl 2024-03-04T16:39:29.853 DEBUG:boto:Data: 2024-03-04T16:39:29.853 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:39:29.853 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.853 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.853 DEBUG:boto:Params: {} 2024-03-04T16:39:29.853 DEBUG:boto:Token: None 2024-03-04T16:39:29.853 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT x-amz-acl:public-read /sbnvju-34/?acl 2024-03-04T16:39:29.853 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:xJpy3Ap27j+7ino8TKYdvT4Tv3g= 2024-03-04T16:39:29.853 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xJpy3Ap27j+7ino8TKYdvT4Tv3g=', 'Content-Length': '0'} 2024-03-04T16:39:29.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000959ad14cf7dd4c41-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.861 DEBUG:boto:path=/sbnvju-34/ 2024-03-04T16:39:29.861 DEBUG:boto:auth_path=/sbnvju-34/ 2024-03-04T16:39:29.861 DEBUG:boto:path=/sbnvju-34/?acl 2024-03-04T16:39:29.861 DEBUG:boto:auth_path=/sbnvju-34/?acl 2024-03-04T16:39:29.861 DEBUG:boto:Method: GET 2024-03-04T16:39:29.861 DEBUG:boto:Path: /sbnvju-34/?acl 2024-03-04T16:39:29.861 DEBUG:boto:Data: 2024-03-04T16:39:29.862 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.862 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.862 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.862 DEBUG:boto:Params: {} 2024-03-04T16:39:29.862 DEBUG:boto:Token: None 2024-03-04T16:39:29.862 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-34/?acl 2024-03-04T16:39:29.862 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:dEBKiEKiZgeiCPKHmt9copsqir8= 2024-03-04T16:39:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dEBKiEKiZgeiCPKHmt9copsqir8=', 'Content-Length': '0'} 2024-03-04T16:39:29.863 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e57f837a17272cce-0065e5f941-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.904 DEBUG:boto:path=/sbnvju-35/ 2024-03-04T16:39:29.904 DEBUG:boto:auth_path=/sbnvju-35/ 2024-03-04T16:39:29.904 DEBUG:boto:path=/sbnvju-35/?acl 2024-03-04T16:39:29.904 DEBUG:boto:auth_path=/sbnvju-35/?acl 2024-03-04T16:39:29.905 DEBUG:boto:Method: GET 2024-03-04T16:39:29.905 DEBUG:boto:Path: /sbnvju-35/?acl 2024-03-04T16:39:29.905 DEBUG:boto:Data: 2024-03-04T16:39:29.905 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.905 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.905 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.905 DEBUG:boto:Params: {} 2024-03-04T16:39:29.905 DEBUG:boto:Token: None 2024-03-04T16:39:29.905 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-35/?acl 2024-03-04T16:39:29.906 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IOvpWtlPtaVbPRyFmDC/fsRavAA= 2024-03-04T16:39:29.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IOvpWtlPtaVbPRyFmDC/fsRavAA=', 'Content-Length': '0'} 2024-03-04T16:39:29.907 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094e6d01d273963a8-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '489'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.907 DEBUG:boto:path=/sbnvju-35/ 2024-03-04T16:39:29.907 DEBUG:boto:auth_path=/sbnvju-35/ 2024-03-04T16:39:29.907 DEBUG:boto:path=/sbnvju-35/?acl 2024-03-04T16:39:29.907 DEBUG:boto:auth_path=/sbnvju-35/?acl 2024-03-04T16:39:29.908 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.908 DEBUG:boto:Path: /sbnvju-35/?acl 2024-03-04T16:39:29.908 DEBUG:boto:Data: 2024-03-04T16:39:29.908 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-03-04T16:39:29.908 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.908 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.908 DEBUG:boto:Params: {} 2024-03-04T16:39:29.908 DEBUG:boto:Token: None 2024-03-04T16:39:29.908 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT x-amz-acl:public-read /sbnvju-35/?acl 2024-03-04T16:39:29.908 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5l09D3i4T8aLZ2jMxl6QeWJvB2s= 2024-03-04T16:39:29.909 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5l09D3i4T8aLZ2jMxl6QeWJvB2s=', 'Content-Length': '0'} 2024-03-04T16:39:29.922 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bc80a7fbd3295163-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.922 DEBUG:boto:path=/sbnvju-35/ 2024-03-04T16:39:29.922 DEBUG:boto:auth_path=/sbnvju-35/ 2024-03-04T16:39:29.922 DEBUG:boto:path=/sbnvju-35/?acl 2024-03-04T16:39:29.922 DEBUG:boto:auth_path=/sbnvju-35/?acl 2024-03-04T16:39:29.922 DEBUG:boto:Method: GET 2024-03-04T16:39:29.922 DEBUG:boto:Path: /sbnvju-35/?acl 2024-03-04T16:39:29.922 DEBUG:boto:Data: 2024-03-04T16:39:29.922 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.923 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.923 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.923 DEBUG:boto:Params: {} 2024-03-04T16:39:29.923 DEBUG:boto:Token: None 2024-03-04T16:39:29.923 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-35/?acl 2024-03-04T16:39:29.923 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IOvpWtlPtaVbPRyFmDC/fsRavAA= 2024-03-04T16:39:29.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IOvpWtlPtaVbPRyFmDC/fsRavAA=', 'Content-Length': '0'} 2024-03-04T16:39:29.924 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021c7b31df46284f8-0065e5f941-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '681'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-03-04T16:39:29.925 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-36 2024-03-04T16:39:29.925 DEBUG:boto:path=/sbnvju-36/ 2024-03-04T16:39:29.925 DEBUG:boto:auth_path=/sbnvju-36/ 2024-03-04T16:39:29.926 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.926 DEBUG:boto:Path: /sbnvju-36/ 2024-03-04T16:39:29.926 DEBUG:boto:Data: 2024-03-04T16:39:29.926 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.926 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.926 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.926 DEBUG:boto:Params: {} 2024-03-04T16:39:29.926 DEBUG:boto:Token: None 2024-03-04T16:39:29.926 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-36/ 2024-03-04T16:39:29.926 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:R/6elVBlz1I3ut49ItU+dzeJr7s= 2024-03-04T16:39:29.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:R/6elVBlz1I3ut49ItU+dzeJr7s=', 'Content-Length': '0'} 2024-03-04T16:39:29.940 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042a8d006a740c059-0065e5f941-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.941 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-37 2024-03-04T16:39:29.941 DEBUG:boto:path=/sbnvju-37/ 2024-03-04T16:39:29.941 DEBUG:boto:auth_path=/sbnvju-37/ 2024-03-04T16:39:29.941 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.941 DEBUG:boto:Path: /sbnvju-37/ 2024-03-04T16:39:29.941 DEBUG:boto:Data: 2024-03-04T16:39:29.941 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.941 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:29.941 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.941 DEBUG:boto:Params: {} 2024-03-04T16:39:29.942 DEBUG:boto:Token: None 2024-03-04T16:39:29.942 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-37/ 2024-03-04T16:39:29.942 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sYevhgIiiVm8WYHjde7N8uHRJVQ= 2024-03-04T16:39:29.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sYevhgIiiVm8WYHjde7N8uHRJVQ=', 'Content-Length': '0'} 2024-03-04T16:39:29.970 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000fbadf062c24db84-0065e5f941-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.970 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-38 2024-03-04T16:39:29.970 DEBUG:boto:path=/sbnvju-38/ 2024-03-04T16:39:29.971 DEBUG:boto:auth_path=/sbnvju-38/ 2024-03-04T16:39:29.971 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.971 DEBUG:boto:Path: /sbnvju-38/ 2024-03-04T16:39:29.971 DEBUG:boto:Data: 2024-03-04T16:39:29.971 DEBUG:boto:Headers: {} 2024-03-04T16:39:29.971 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:29.971 DEBUG:boto:Port: 8001 2024-03-04T16:39:29.971 DEBUG:boto:Params: {} 2024-03-04T16:39:29.971 DEBUG:boto:Token: None 2024-03-04T16:39:29.971 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-38/ 2024-03-04T16:39:29.972 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:W3IpEuYrml9EvXhILnSXcD/Ntkk= 2024-03-04T16:39:29.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:W3IpEuYrml9EvXhILnSXcD/Ntkk=', 'Content-Length': '0'} 2024-03-04T16:39:29.994 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5cd8bb73bed17cf-0065e5f941-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:29.995 DEBUG:boto:path=/sbnvju-36/ 2024-03-04T16:39:29.995 DEBUG:boto:auth_path=/sbnvju-36/ 2024-03-04T16:39:29.995 DEBUG:boto:path=/sbnvju-36/?cors 2024-03-04T16:39:29.995 DEBUG:boto:auth_path=/sbnvju-36/?cors 2024-03-04T16:39:29.995 DEBUG:boto:Method: PUT 2024-03-04T16:39:29.995 DEBUG:boto:Path: /sbnvju-36/?cors 2024-03-04T16:39:29.995 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T16:39:29.995 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T16:39:29.995 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:29.995 DEBUG:boto:Port: 8000 2024-03-04T16:39:29.996 DEBUG:boto:Params: {} 2024-03-04T16:39:29.996 DEBUG:boto:Token: None 2024-03-04T16:39:29.996 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 16:39:29 GMT /sbnvju-36/?cors 2024-03-04T16:39:29.996 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:/mly42PCmN08iMwZKnf1mK3POJM= 2024-03-04T16:39:29.996 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:29 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/mly42PCmN08iMwZKnf1mK3POJM=', 'Content-Length': '218'} 2024-03-04T16:39:30.001 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000036be1fbb1d329880-0065e5f942-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:30.001 DEBUG:boto:path=/sbnvju-36/ 2024-03-04T16:39:30.001 DEBUG:boto:auth_path=/sbnvju-36/ 2024-03-04T16:39:30.001 DEBUG:boto:path=/sbnvju-36/?cors 2024-03-04T16:39:30.001 DEBUG:boto:auth_path=/sbnvju-36/?cors 2024-03-04T16:39:30.001 DEBUG:boto:Method: GET 2024-03-04T16:39:30.002 DEBUG:boto:Path: /sbnvju-36/?cors 2024-03-04T16:39:30.002 DEBUG:boto:Data: 2024-03-04T16:39:30.002 DEBUG:boto:Headers: {} 2024-03-04T16:39:30.002 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:30.002 DEBUG:boto:Port: 8000 2024-03-04T16:39:30.002 DEBUG:boto:Params: {} 2024-03-04T16:39:30.002 DEBUG:boto:Token: None 2024-03-04T16:39:30.002 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:30 GMT /sbnvju-36/?cors 2024-03-04T16:39:30.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aHEQzrKpRfs2nyDDN97mcijpxLs= 2024-03-04T16:39:30.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHEQzrKpRfs2nyDDN97mcijpxLs=', 'Content-Length': '0'} 2024-03-04T16:39:30.004 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059cd8fcba60257ae-0065e5f942-4341-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 16:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:30.044 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T16:39:30.045 DEBUG:boto:path=/sbnvju-37/ 2024-03-04T16:39:30.045 DEBUG:boto:auth_path=/sbnvju-37/ 2024-03-04T16:39:30.045 DEBUG:boto:path=/sbnvju-37/?cors 2024-03-04T16:39:30.045 DEBUG:boto:auth_path=/sbnvju-37/?cors 2024-03-04T16:39:30.045 DEBUG:boto:Method: PUT 2024-03-04T16:39:30.045 DEBUG:boto:Path: /sbnvju-37/?cors 2024-03-04T16:39:30.045 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T16:39:30.045 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T16:39:30.045 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:30.045 DEBUG:boto:Port: 8000 2024-03-04T16:39:30.045 DEBUG:boto:Params: {} 2024-03-04T16:39:30.046 DEBUG:boto:Token: None 2024-03-04T16:39:30.046 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 16:39:30 GMT /sbnvju-37/?cors 2024-03-04T16:39:30.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Fpx03VPTpgO+MJ16ION62zGoUMA= 2024-03-04T16:39:30.046 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Fpx03VPTpgO+MJ16ION62zGoUMA=', 'Content-Length': '218'} 2024-03-04T16:39:31.061 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab09b564e8a3dad3-0065e5f942-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:31.061 DEBUG:boto:path=/sbnvju-37/ 2024-03-04T16:39:31.062 DEBUG:boto:auth_path=/sbnvju-37/ 2024-03-04T16:39:31.062 DEBUG:boto:path=/sbnvju-37/?cors 2024-03-04T16:39:31.062 DEBUG:boto:auth_path=/sbnvju-37/?cors 2024-03-04T16:39:31.062 DEBUG:boto:Method: GET 2024-03-04T16:39:31.062 DEBUG:boto:Path: /sbnvju-37/?cors 2024-03-04T16:39:31.062 DEBUG:boto:Data: 2024-03-04T16:39:31.062 DEBUG:boto:Headers: {} 2024-03-04T16:39:31.062 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:31.062 DEBUG:boto:Port: 8000 2024-03-04T16:39:31.062 DEBUG:boto:Params: {} 2024-03-04T16:39:31.062 DEBUG:boto:Token: None 2024-03-04T16:39:31.063 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:31 GMT /sbnvju-37/?cors 2024-03-04T16:39:31.063 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bZIW4NaJq9HKkRbqZ3rcQkB2+es= 2024-03-04T16:39:31.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bZIW4NaJq9HKkRbqZ3rcQkB2+es=', 'Content-Length': '0'} 2024-03-04T16:39:31.064 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6ed9037d17acef4-0065e5f943-4307-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 16:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:31.064 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T16:39:31.064 DEBUG:boto:path=/sbnvju-38/ 2024-03-04T16:39:31.065 DEBUG:boto:auth_path=/sbnvju-38/ 2024-03-04T16:39:31.065 DEBUG:boto:path=/sbnvju-38/?cors 2024-03-04T16:39:31.065 DEBUG:boto:auth_path=/sbnvju-38/?cors 2024-03-04T16:39:31.065 DEBUG:boto:Method: PUT 2024-03-04T16:39:31.065 DEBUG:boto:Path: /sbnvju-38/?cors 2024-03-04T16:39:31.065 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-03-04T16:39:31.065 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-03-04T16:39:31.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:31.065 DEBUG:boto:Port: 8001 2024-03-04T16:39:31.065 DEBUG:boto:Params: {} 2024-03-04T16:39:31.066 DEBUG:boto:Token: None 2024-03-04T16:39:31.066 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 04 Mar 2024 16:39:31 GMT /sbnvju-38/?cors 2024-03-04T16:39:31.066 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PpxAK3V99KEghH77BscA8+zo8IM= 2024-03-04T16:39:31.066 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PpxAK3V99KEghH77BscA8+zo8IM=', 'Content-Length': '218'} 2024-03-04T16:39:32.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d2feb24552df53a-0065e5f943-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:32.077 DEBUG:boto:path=/sbnvju-38/ 2024-03-04T16:39:32.077 DEBUG:boto:auth_path=/sbnvju-38/ 2024-03-04T16:39:32.077 DEBUG:boto:path=/sbnvju-38/?cors 2024-03-04T16:39:32.077 DEBUG:boto:auth_path=/sbnvju-38/?cors 2024-03-04T16:39:32.077 DEBUG:boto:Method: GET 2024-03-04T16:39:32.077 DEBUG:boto:Path: /sbnvju-38/?cors 2024-03-04T16:39:32.078 DEBUG:boto:Data: 2024-03-04T16:39:32.078 DEBUG:boto:Headers: {} 2024-03-04T16:39:32.078 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:32.078 DEBUG:boto:Port: 8001 2024-03-04T16:39:32.078 DEBUG:boto:Params: {} 2024-03-04T16:39:32.078 DEBUG:boto:Token: None 2024-03-04T16:39:32.078 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 16:39:32 GMT /sbnvju-38/?cors 2024-03-04T16:39:32.078 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NUcIB6ThbG8ZxElVQwPxNDZaBVY= 2024-03-04T16:39:32.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NUcIB6ThbG8ZxElVQwPxNDZaBVY=', 'Content-Length': '0'} 2024-03-04T16:39:32.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5242996fdbfa55a-0065e5f944-4348-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:32.120 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-03-04T16:39:32.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-03-04T16:39:32.121 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-39 2024-03-04T16:39:32.121 DEBUG:boto:path=/sbnvju-39/ 2024-03-04T16:39:32.121 DEBUG:boto:auth_path=/sbnvju-39/ 2024-03-04T16:39:32.121 DEBUG:boto:Method: PUT 2024-03-04T16:39:32.121 DEBUG:boto:Path: /sbnvju-39/ 2024-03-04T16:39:32.121 DEBUG:boto:Data: 2024-03-04T16:39:32.122 DEBUG:boto:Headers: {} 2024-03-04T16:39:32.122 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:39:32.122 DEBUG:boto:Port: 8000 2024-03-04T16:39:32.122 DEBUG:boto:Params: {} 2024-03-04T16:39:32.122 DEBUG:boto:Token: None 2024-03-04T16:39:32.122 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:32 GMT /sbnvju-39/ 2024-03-04T16:39:32.122 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:njY7fQZ7hrtv6x9NwUhR1T3/hqg= 2024-03-04T16:39:32.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:njY7fQZ7hrtv6x9NwUhR1T3/hqg=', 'Content-Length': '0'} 2024-03-04T16:39:32.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df1b100fd70e49f6-0065e5f944-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:32.162 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-40 2024-03-04T16:39:32.162 DEBUG:boto:path=/sbnvju-40/ 2024-03-04T16:39:32.162 DEBUG:boto:auth_path=/sbnvju-40/ 2024-03-04T16:39:32.162 DEBUG:boto:Method: PUT 2024-03-04T16:39:32.163 DEBUG:boto:Path: /sbnvju-40/ 2024-03-04T16:39:32.163 DEBUG:boto:Data: 2024-03-04T16:39:32.163 DEBUG:boto:Headers: {} 2024-03-04T16:39:32.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:39:32.163 DEBUG:boto:Port: 8000 2024-03-04T16:39:32.163 DEBUG:boto:Params: {} 2024-03-04T16:39:32.163 DEBUG:boto:Token: None 2024-03-04T16:39:32.163 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:32 GMT /sbnvju-40/ 2024-03-04T16:39:32.163 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IDCxGACbsbOy3+ceit0WCPH+9OE= 2024-03-04T16:39:32.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IDCxGACbsbOy3+ceit0WCPH+9OE=', 'Content-Length': '0'} 2024-03-04T16:39:32.190 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e3b3f910ea93bd0-0065e5f944-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:32.191 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-41 2024-03-04T16:39:32.191 DEBUG:boto:path=/sbnvju-41/ 2024-03-04T16:39:32.191 DEBUG:boto:auth_path=/sbnvju-41/ 2024-03-04T16:39:32.191 DEBUG:boto:Method: PUT 2024-03-04T16:39:32.191 DEBUG:boto:Path: /sbnvju-41/ 2024-03-04T16:39:32.191 DEBUG:boto:Data: 2024-03-04T16:39:32.191 DEBUG:boto:Headers: {} 2024-03-04T16:39:32.191 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:39:32.191 DEBUG:boto:Port: 8001 2024-03-04T16:39:32.191 DEBUG:boto:Params: {} 2024-03-04T16:39:32.192 DEBUG:boto:Token: None 2024-03-04T16:39:32.192 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:39:32 GMT /sbnvju-41/ 2024-03-04T16:39:32.192 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rcFOlF05vUqPmBJWvb8hZwOF1K0= 2024-03-04T16:39:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rcFOlF05vUqPmBJWvb8hZwOF1K0=', 'Content-Length': '0'} 2024-03-04T16:39:32.216 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2fe79137587c585-0065e5f944-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:39:32.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:32.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:32.216 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:39:32.382 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570372.221520_469.1","last_update":"2024-03-04T16:39:32.221520Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:39:32.383 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:32.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:39:32.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:32.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:32.384 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:32.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:32.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:32.474 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:32.488+0000 7f5c84a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:32.475 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:32.489+0000 7f5c84a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:32.493 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:32Z"} 2024-03-04T16:39:32.494 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:32Z"} 2024-03-04T16:39:32.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:32.494 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:32.494 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:32.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:39:37.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:37.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:37.496 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:37.542 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:37.542 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:37.585 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:37.599+0000 7f608eec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:37.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:37.600+0000 7f608eec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:37.604 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:37Z"} 2024-03-04T16:39:37.604 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:37Z"} 2024-03-04T16:39:37.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:37.605 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:37.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:37.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T16:39:42.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:42.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:42.606 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:42.653 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:42.653 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:42.696 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:42.710+0000 7f568cf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:42.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:42.711+0000 7f568cf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:42.715 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:42Z"} 2024-03-04T16:39:42.716 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:42Z"} 2024-03-04T16:39:42.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:42.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:42.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:42.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:39:47.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:47.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:47.718 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:47.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:47.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:47.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:47.823+0000 7f41a4f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:47.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:47.824+0000 7f41a4f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:47.829 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:47Z"} 2024-03-04T16:39:47.830 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:47Z"} 2024-03-04T16:39:47.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:47.830 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:47.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:47.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:39:52.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:52.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:52.831 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:52.879 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:52.879 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:52.978 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:52.993+0000 7fd30b34b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:52.979 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:52.994+0000 7fd30b34b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:52.998 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:52Z"} 2024-03-04T16:39:52.999 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:52Z"} 2024-03-04T16:39:52.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:52.999 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:52.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:52.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T16:39:58.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:58.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:39:58.001 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:39:58.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:39:58.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:39:58.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:58.110+0000 7f7748730800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:58.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:39:58.111+0000 7f7748730800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:39:58.115 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:58Z"} 2024-03-04T16:39:58.115 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:58Z"} 2024-03-04T16:39:58.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:39:58.115 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:58.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:39:58.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:40:03.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:03.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:03.117 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:03.222+0000 7ff6cf814800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:03.223+0000 7ff6cf814800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:03.227 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:03Z"} 2024-03-04T16:40:03.227 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:03Z"} 2024-03-04T16:40:03.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:03.228 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:03.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:03.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:40:08.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:08.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:08.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:08.278 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:08.278 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:08.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:08.337+0000 7f747873b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:08.323 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:08.338+0000 7f747873b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:08.342 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:08Z"} 2024-03-04T16:40:08.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:08Z"} 2024-03-04T16:40:08.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:08.343 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:08.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:08.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T16:40:13.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:13.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:13.344 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:13.404 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:13.405 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:13.511 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:13.526+0000 7fe5b7444800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:13.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:13.527+0000 7fe5b7444800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:13.532 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:13Z"} 2024-03-04T16:40:13.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:13Z"} 2024-03-04T16:40:13.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:13.532 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:13.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:13.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.795745_422.1 2024-03-04T16:40:18.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:18.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:18.534 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:18.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:18.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:18.627 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:18.642+0000 7fab54922800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:18.628 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:18.643+0000 7fab54922800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:18.647 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:18Z"} 2024-03-04T16:40:18.647 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:18Z"} 2024-03-04T16:40:18.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:18.647 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:18.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:18.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T16:40:23.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:23.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:23.649 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:23.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:23.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:23.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:23.765+0000 7fa7b6439800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:23.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:23.766+0000 7fa7b6439800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:23.771 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:23Z"} 2024-03-04T16:40:23.771 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:23Z"} 2024-03-04T16:40:23.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:23.772 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:23.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:23.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.647116_415.1 2024-03-04T16:40:28.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:28.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:28.773 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:28.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:28.828 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:28.876 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:28.890+0000 7fb18a012800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:28.876 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:28.891+0000 7fb18a012800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:28.897 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:28Z"} 2024-03-04T16:40:28.897 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:28Z"} 2024-03-04T16:40:28.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:28.898 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:28.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:28.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:40:33.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:33.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:33.900 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:33.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:33.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:34.017+0000 7f5075faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:34.018+0000 7f5075faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:34.023 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:34Z"} 2024-03-04T16:40:34.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:34Z"} 2024-03-04T16:40:34.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.819018_429.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.819018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:34.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:34.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.819018_429.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:34.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.819018_429.1 2024-03-04T16:40:39.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:39.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:39.026 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:39.108 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:39.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:39.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:39.180+0000 7f2253c42800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:39.166 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:39.181+0000 7f2253c42800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:39.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:39Z"} 2024-03-04T16:40:39.186 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:39Z"} 2024-03-04T16:40:39.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:39.186 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:39.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:39.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.838183_430.1 2024-03-04T16:40:44.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:44.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:44.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:44.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:44.236 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:44.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:44.293+0000 7ffb18010800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:44.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:44.294+0000 7ffb18010800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:44.298 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:44Z"} 2024-03-04T16:40:44.298 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:44Z"} 2024-03-04T16:40:44.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:44.299 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:44.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:44.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:40:49.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:49.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:49.300 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:49.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:49.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:49.393 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:49.408+0000 7fe705326800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:49.394 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:49.409+0000 7fe705326800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:49.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:49Z"} 2024-03-04T16:40:49.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:49Z"} 2024-03-04T16:40:49.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:49.414 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:49.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:49.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T16:40:54.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:54.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:54.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:54.465 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:54.465 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:54.523+0000 7fc1ea711800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:54.524+0000 7fc1ea711800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:54.527 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:54Z"} 2024-03-04T16:40:54.527 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:54Z"} 2024-03-04T16:40:54.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:54.528 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:54.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:54.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.816633_428.1 2024-03-04T16:40:59.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:59.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:40:59.530 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:40:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:40:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:40:59.631 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:59.646+0000 7fefca7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:59.632 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:40:59.647+0000 7fefca7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:40:59.652 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:59Z"} 2024-03-04T16:40:59.652 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:59Z"} 2024-03-04T16:40:59.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:40:59.653 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:59.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:40:59.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.662426_417.1 2024-03-04T16:41:04.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:04.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:04.654 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:04.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:04.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:04.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:04.760+0000 7f5761b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:04.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:04.761+0000 7f5761b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:04.764 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:04Z"} 2024-03-04T16:41:04.764 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:04Z"} 2024-03-04T16:41:04.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:04.765 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:04.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:04.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:41:09.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:09.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:09.766 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:09.812 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:09.812 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:09.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:09.872+0000 7ff4f3232800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:09.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:09.873+0000 7ff4f3232800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:09.876 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:09Z"} 2024-03-04T16:41:09.877 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:09Z"} 2024-03-04T16:41:09.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:09.877 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:09.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:09.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:41:14.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:14.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:14.878 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:14.926 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:14.926 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:14.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:14.984+0000 7fa95c9af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:14.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:14.985+0000 7fa95c9af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:14.989 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:14Z"} 2024-03-04T16:41:14.989 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:14Z"} 2024-03-04T16:41:14.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:14.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:14.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:14.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T16:41:19.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:19.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:19.992 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:20.046 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:20.046 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:20.103 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:20.118+0000 7f461b224800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:20.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:20.119+0000 7f461b224800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:20.123 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:20Z"} 2024-03-04T16:41:20.123 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:20Z"} 2024-03-04T16:41:20.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:20.124 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:20.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:20.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T16:41:25.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:25.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:25.125 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:25.178 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:25.178 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:25.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:25.241+0000 7fad49011800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:25.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:25.242+0000 7fad49011800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:25.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:25Z"} 2024-03-04T16:41:25.247 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:25Z"} 2024-03-04T16:41:25.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:25.247 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:25.247 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:25.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.711729_418.1 2024-03-04T16:41:30.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:30.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:30.249 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:30.325 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:30.325 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:30.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:30.429+0000 7f4b19a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:30.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:30.430+0000 7f4b19a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:30.434 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:30Z"} 2024-03-04T16:41:30.434 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:30Z"} 2024-03-04T16:41:30.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:30.434 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:30.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:30.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.643873_413.1 2024-03-04T16:41:35.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:35.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:35.436 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:35.482 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:35.482 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:35.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:35.540+0000 7f50d4024800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:35.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:35.541+0000 7f50d4024800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:35.544 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:35Z"} 2024-03-04T16:41:35.545 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:35Z"} 2024-03-04T16:41:35.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:35.545 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:35.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:35.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.715026_419.1 2024-03-04T16:41:40.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:40.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:40.547 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:40.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:40.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:40.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:40.660+0000 7fb1aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:40.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:40.661+0000 7fb1aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:40.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:40Z"} 2024-03-04T16:41:40.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:40Z"} 2024-03-04T16:41:40.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:40.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:40.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:40.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T16:41:45.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:45.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:45.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:45.769 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:45.784+0000 7fe97fc9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:45.769 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:45.785+0000 7fe97fc9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:45.789 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:45Z"} 2024-03-04T16:41:45.790 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:45Z"} 2024-03-04T16:41:45.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:45.790 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:45.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:45.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:41:50.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:50.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:50.793 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:50.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:50.846 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:50.909+0000 7f0367f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:50.910+0000 7f0367f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:50.915 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:50Z"} 2024-03-04T16:41:50.915 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:50Z"} 2024-03-04T16:41:50.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:50.915 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:50.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:50.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T16:41:55.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:55.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:41:55.917 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:41:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:41:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:41:56.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:56.034+0000 7fd06868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:56.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:41:56.035+0000 7fd06868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:41:56.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:56Z"} 2024-03-04T16:41:56.039 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:56Z"} 2024-03-04T16:41:56.039 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:41:56.039 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:56.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:41:56.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.643873_413.1 2024-03-04T16:42:01.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:01.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:01.041 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:01.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:01.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:01.145 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:01.161+0000 7f15dc20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:01.146 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:01.162+0000 7f15dc20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:01.167 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:01Z"} 2024-03-04T16:42:01.167 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:01Z"} 2024-03-04T16:42:01.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:01.167 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:01.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:01.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.738966_421.1 2024-03-04T16:42:06.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:06.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:06.169 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:06.275+0000 7f6c7a40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:06.276+0000 7f6c7a40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:06.280 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:06Z"} 2024-03-04T16:42:06.280 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:06Z"} 2024-03-04T16:42:06.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:06.280 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:06.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:06.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:42:11.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:11.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:11.282 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:11.330 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:11.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:11.389+0000 7f84e8e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:11.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:11.389+0000 7f84e8e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:11.392 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:11Z"} 2024-03-04T16:42:11.392 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:11Z"} 2024-03-04T16:42:11.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:11.393 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:11.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:11.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.711729_418.1 2024-03-04T16:42:16.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:16.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:16.395 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:16.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:16.443 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:16.486 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:16.502+0000 7f189713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:16.487 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:16.503+0000 7f189713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:16.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:16Z"} 2024-03-04T16:42:16.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:16Z"} 2024-03-04T16:42:16.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:16.507 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:16.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:16.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:42:21.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:21.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:21.508 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:21.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:21.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:21.610 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:21.626+0000 7fad53c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:21.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:21.627+0000 7fad53c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:21.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:21Z"} 2024-03-04T16:42:21.631 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:21Z"} 2024-03-04T16:42:21.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:21.631 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:21.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:21.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:42:26.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:26.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:26.633 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:26.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:26.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:26.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:26.752+0000 7fe101824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:26.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:26.753+0000 7fe101824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:26.758 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:26Z"} 2024-03-04T16:42:26.758 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:26Z"} 2024-03-04T16:42:26.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:26.759 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:26.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:26.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:42:31.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:31.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:31.760 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:31.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:31.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:31.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:31.880+0000 7f649631e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:31.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:31.881+0000 7f649631e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:31.885 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:31Z"} 2024-03-04T16:42:31.885 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:31Z"} 2024-03-04T16:42:31.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:31.885 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:31.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:31.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T16:42:36.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:36.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:36.887 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:36.968 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:36.968 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:37.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:37.030+0000 7fa434b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:37.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:37.031+0000 7fa434b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:37.034 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:37Z"} 2024-03-04T16:42:37.035 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:37Z"} 2024-03-04T16:42:37.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:37.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:37.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:37.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T16:42:42.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:42.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:42.037 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:42.083 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:42.083 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:42.142+0000 7f0231a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:42.126 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:42.142+0000 7f0231a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:42.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:42Z"} 2024-03-04T16:42:42.145 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:42Z"} 2024-03-04T16:42:42.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:42.146 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:42.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:42.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:42:47.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:47.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:47.148 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:47.197 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:47.197 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:47.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:47.259+0000 7fde4cc85800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:47.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:47.260+0000 7fde4cc85800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:47.263 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:47Z"} 2024-03-04T16:42:47.264 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:47Z"} 2024-03-04T16:42:47.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:47.264 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:47.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:47.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T16:42:52.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:52.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:52.265 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:52.318 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:52.318 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:52.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:52.383+0000 7fc4c6b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:52.367 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:52.384+0000 7fc4c6b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:52.388 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:52Z"} 2024-03-04T16:42:52.388 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:52Z"} 2024-03-04T16:42:52.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:52.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:52.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:52.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:42:57.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:57.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:42:57.390 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:42:57.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:42:57.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:42:57.489 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:57.506+0000 7fb698f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:57.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:42:57.507+0000 7fb698f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:42:57.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:57Z"} 2024-03-04T16:42:57.510 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:57Z"} 2024-03-04T16:42:57.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:42:57.511 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:57.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:42:57.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T16:43:02.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:02.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:02.513 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:02.592 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:02.593 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:02.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:02.661+0000 7f5582ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:02.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:02.661+0000 7f5582ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:02.665 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:02Z"} 2024-03-04T16:43:02.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:02Z"} 2024-03-04T16:43:02.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:02.665 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:02.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:02.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T16:43:07.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:07.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:07.667 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:07.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:07.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:07.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:07.776+0000 7f663ff3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:07.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:07.777+0000 7f663ff3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:07.779 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:07Z"} 2024-03-04T16:43:07.780 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:07Z"} 2024-03-04T16:43:07.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:07.780 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:07.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:07.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T16:43:12.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:12.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:12.782 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:12.832 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:12.832 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:12.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:12.895+0000 7f191d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:12.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:12.896+0000 7f191d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:12.899 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:12Z"} 2024-03-04T16:43:12.899 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:12Z"} 2024-03-04T16:43:12.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:12.900 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:12.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:12.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T16:43:17.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:17.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:17.902 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:17.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:17.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:18.001 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:18.018+0000 7f28b2412800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:18.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:18.019+0000 7f28b2412800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:18.022 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:18Z"} 2024-03-04T16:43:18.022 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:18Z"} 2024-03-04T16:43:18.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:18.023 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:18.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:18.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T16:43:23.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:23.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:23.025 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:23.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:23.111 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:23.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:23.174+0000 7fe1fe6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:23.174+0000 7fe1fe6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:23.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:23Z"} 2024-03-04T16:43:23.179 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:23Z"} 2024-03-04T16:43:23.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:23.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:23.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:23.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:43:28.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:28.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:28.182 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:28.272 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:28.289+0000 7fe2ad726800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:28.273 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:28.290+0000 7fe2ad726800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:28.292 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:28Z"} 2024-03-04T16:43:28.293 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:28Z"} 2024-03-04T16:43:28.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:28.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:28.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:28.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T16:43:33.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:33.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:33.294 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:33.401+0000 7fcce0230800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:33.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:33.402+0000 7fcce0230800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:33.404 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:33Z"} 2024-03-04T16:43:33.404 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:33Z"} 2024-03-04T16:43:33.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:33.404 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:33.404 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:33.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:43:38.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:38.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:38.406 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:38.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:38.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:38.515+0000 7faefe730800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:38.516+0000 7faefe730800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:38.519 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:38Z"} 2024-03-04T16:43:38.519 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:38Z"} 2024-03-04T16:43:38.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:38.520 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:38.520 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:38.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T16:43:43.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:43.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:43.521 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:43.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:43.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:43.640+0000 7f858413c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:43.641+0000 7f858413c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:43Z"} 2024-03-04T16:43:43.644 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:43Z"} 2024-03-04T16:43:43.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:43.645 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:43.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:43.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T16:43:48.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:48.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:48.647 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:48.699 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:48.699 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:48.770 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:48.787+0000 7f5fa412a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:48.771 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:48.788+0000 7f5fa412a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:48.792 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:48Z"} 2024-03-04T16:43:48.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:48Z"} 2024-03-04T16:43:48.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.815644_427.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.815644Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:48.792 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:48.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.815644_427.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:48.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.815644_427.1 2024-03-04T16:43:53.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:53.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:53.793 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:53.847 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:53.847 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:53.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:53.983+0000 7f050b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:53.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:53.984+0000 7f050b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:53.988 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:53Z"} 2024-03-04T16:43:53.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:53Z"} 2024-03-04T16:43:53.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:53.988 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:53.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:53.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T16:43:58.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:58.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:43:58.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:43:59.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:43:59.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:43:59.080 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:59.097+0000 7f449fb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:59.081 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:43:59.098+0000 7f449fb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:43:59.100 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:59Z"} 2024-03-04T16:43:59.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:59Z"} 2024-03-04T16:43:59.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:43:59.101 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:59.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:43:59.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.736351_420.1 2024-03-04T16:44:04.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:04.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:04.102 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:04.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:04.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:04.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:04.209+0000 7fb2e4f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:04.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:04.210+0000 7fb2e4f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:04.212 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:04Z"} 2024-03-04T16:44:04.212 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:04Z"} 2024-03-04T16:44:04.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:04.213 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:04.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:04.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T16:44:09.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:09.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:09.215 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:09.267 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:09.267 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:09.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:09.333+0000 7f233c32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:09.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:09.334+0000 7f233c32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:09.336 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:09Z"} 2024-03-04T16:44:09.337 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:09Z"} 2024-03-04T16:44:09.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:09.337 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:09.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:09.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T16:44:14.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:14.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:14.339 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:14.437 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:14.454+0000 7f49e77af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:14.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:14.455+0000 7f49e77af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:14.458 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:14Z"} 2024-03-04T16:44:14.458 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:14Z"} 2024-03-04T16:44:14.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:14.458 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:14.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:14.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.662426_417.1 2024-03-04T16:44:19.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:19.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:19.460 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:19.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:19.513 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:19.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:19.580+0000 7f7de8c20800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:19.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:19.581+0000 7f7de8c20800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:19.582 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:19Z"} 2024-03-04T16:44:19.583 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:19Z"} 2024-03-04T16:44:19.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:19.583 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:19.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:19.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T16:44:24.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:24.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:24.585 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:24.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:24.748+0000 7fd3e9e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:24.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:24.749+0000 7fd3e9e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:24.751 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:24Z"} 2024-03-04T16:44:24.752 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:24Z"} 2024-03-04T16:44:24.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:24.752 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:24.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:24.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T16:44:29.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:29.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:29.753 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:29.800 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:29.800 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:29.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:29.863+0000 7fc5d6a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:29.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:29.864+0000 7fc5d6a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:29Z"} 2024-03-04T16:44:29.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:29Z"} 2024-03-04T16:44:29.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:29.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:29.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:29.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T16:44:34.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:34.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:34.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:34.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:34.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:34.976+0000 7fe017617800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:34.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:34.977+0000 7fe017617800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:34.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:34Z"} 2024-03-04T16:44:34.979 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:34Z"} 2024-03-04T16:44:34.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:34.979 DEBUG:rgw_multi.tests:log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:34.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:34.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T16:44:39.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-03-04T16:44:39.983 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-42 2024-03-04T16:44:39.983 DEBUG:boto:path=/sbnvju-42/ 2024-03-04T16:44:39.983 DEBUG:boto:auth_path=/sbnvju-42/ 2024-03-04T16:44:39.984 DEBUG:boto:Method: PUT 2024-03-04T16:44:39.984 DEBUG:boto:Path: /sbnvju-42/ 2024-03-04T16:44:39.984 DEBUG:boto:Data: 2024-03-04T16:44:39.984 DEBUG:boto:Headers: {} 2024-03-04T16:44:39.984 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:44:39.984 DEBUG:boto:Port: 8000 2024-03-04T16:44:39.984 DEBUG:boto:Params: {} 2024-03-04T16:44:39.985 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:44:39.985 DEBUG:boto:Token: None 2024-03-04T16:44:39.985 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:44:39 GMT /sbnvju-42/ 2024-03-04T16:44:39.985 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vjlv8lW59s5m9USSrLigp5JjxqY= 2024-03-04T16:44:39.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vjlv8lW59s5m9USSrLigp5JjxqY=', 'Content-Length': '0'} 2024-03-04T16:44:40.008 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079b5cee737700a81-0065e5fa78-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:44:40.009 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-43 2024-03-04T16:44:40.009 DEBUG:boto:path=/sbnvju-43/ 2024-03-04T16:44:40.009 DEBUG:boto:auth_path=/sbnvju-43/ 2024-03-04T16:44:40.009 DEBUG:boto:Method: PUT 2024-03-04T16:44:40.009 DEBUG:boto:Path: /sbnvju-43/ 2024-03-04T16:44:40.009 DEBUG:boto:Data: 2024-03-04T16:44:40.009 DEBUG:boto:Headers: {} 2024-03-04T16:44:40.009 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:44:40.010 DEBUG:boto:Port: 8000 2024-03-04T16:44:40.010 DEBUG:boto:Params: {} 2024-03-04T16:44:40.010 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:44:40.010 DEBUG:boto:Token: None 2024-03-04T16:44:40.010 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:44:40 GMT /sbnvju-43/ 2024-03-04T16:44:40.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oY0QyuqSlHy133iNVpfzhClbebY= 2024-03-04T16:44:40.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oY0QyuqSlHy133iNVpfzhClbebY=', 'Content-Length': '0'} 2024-03-04T16:44:40.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000acd1b1fd42b4a896-0065e5fa78-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:44:40.045 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-44 2024-03-04T16:44:40.045 DEBUG:boto:path=/sbnvju-44/ 2024-03-04T16:44:40.045 DEBUG:boto:auth_path=/sbnvju-44/ 2024-03-04T16:44:40.045 DEBUG:boto:Method: PUT 2024-03-04T16:44:40.045 DEBUG:boto:Path: /sbnvju-44/ 2024-03-04T16:44:40.045 DEBUG:boto:Data: 2024-03-04T16:44:40.046 DEBUG:boto:Headers: {} 2024-03-04T16:44:40.046 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:44:40.046 DEBUG:boto:Port: 8001 2024-03-04T16:44:40.046 DEBUG:boto:Params: {} 2024-03-04T16:44:40.046 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:44:40.046 DEBUG:boto:Token: None 2024-03-04T16:44:40.046 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:44:40 GMT /sbnvju-44/ 2024-03-04T16:44:40.046 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Zr/X+YXo5tX5rxrpCXXd7CyYZ/Y= 2024-03-04T16:44:40.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Zr/X+YXo5tX5rxrpCXXd7CyYZ/Y=', 'Content-Length': '0'} 2024-03-04T16:44:40.088 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007957b4da74e7db5e-0065e5fa78-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:44:40.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:40.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:40.088 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:44:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570680.085880_523.1","last_update":"2024-03-04T16:44:40.085880Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:44:40.211 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:40.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:44:40.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:40.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:40.212 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:40.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:40.321+0000 7f6f9dd0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:40.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:40.322+0000 7f6f9dd0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:40.324 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:40Z"} 2024-03-04T16:44:40.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:40Z"} 2024-03-04T16:44:40.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:40.324 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:40.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:40.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T16:44:45.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:45.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:45.326 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:45.371 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:45.372 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:45.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:45.514+0000 7f4e1cd2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:45.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:45.516+0000 7f4e1cd2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:45.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:45Z"} 2024-03-04T16:44:45.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:45Z"} 2024-03-04T16:44:45.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:45.529 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:45.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:45.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T16:44:50.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:50.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:50.531 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:50.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:50.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:50.666 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:50.683+0000 7fc71051f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:50.666 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:50.684+0000 7fc71051f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:50.686 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:50Z"} 2024-03-04T16:44:50.687 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:50Z"} 2024-03-04T16:44:50.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:50.687 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:50.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:50.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:44:55.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:55.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:44:55.689 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:44:55.734 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:44:55.734 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:44:55.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:55.795+0000 7f260211f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:55.778 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:44:55.796+0000 7f260211f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:44:55.798 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:55Z"} 2024-03-04T16:44:55.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:55Z"} 2024-03-04T16:44:55.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:44:55.798 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:55.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:44:55.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T16:45:00.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:00.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:00.800 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:00.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:00.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:00.892 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:00.909+0000 7f9654d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:00.893 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:00.910+0000 7f9654d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:00.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:00Z"} 2024-03-04T16:45:00.978 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:00Z"} 2024-03-04T16:45:00.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:00.979 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:00.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:00.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T16:45:05.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:05.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:05.980 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:06.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:06.091+0000 7fbd3042d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:06.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:06.092+0000 7fbd3042d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:06.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:06Z"} 2024-03-04T16:45:06.093 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:06Z"} 2024-03-04T16:45:06.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:06.094 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:06.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:06.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T16:45:11.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:11.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:11.095 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:11.154 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:11.154 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:11.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:11.283+0000 7f5921e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:11.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:11.284+0000 7f5921e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:11.287 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:11Z"} 2024-03-04T16:45:11.287 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:11Z"} 2024-03-04T16:45:11.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:11.288 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:11.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:11.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T16:45:16.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:16.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:16.290 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:16.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:16.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:16.379 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:16.398+0000 7f4ba3f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:16.380 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:16.398+0000 7f4ba3f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:16.399 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:16Z"} 2024-03-04T16:45:16.400 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:16Z"} 2024-03-04T16:45:16.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:16.400 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:16.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:16.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.816633_428.1 2024-03-04T16:45:21.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:21.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:21.402 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:21.449 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:21.449 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:21.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:21.510+0000 7f4474412800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:21.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:21.511+0000 7f4474412800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:21.512 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:21Z"} 2024-03-04T16:45:21.513 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:21Z"} 2024-03-04T16:45:21.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:21.513 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:21.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:21.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:45:26.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:26.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:26.514 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:26.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:26.563 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:26.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:26.624+0000 7f16742c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:26.607 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:26.625+0000 7f16742c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:26.633 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:26Z"} 2024-03-04T16:45:26.633 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:26Z"} 2024-03-04T16:45:26.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:26.634 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:26.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:26.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.801670_425.1 2024-03-04T16:45:31.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:31.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:31.635 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:31.688 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:31.688 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:31.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:31.756+0000 7f46bfb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:31.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:31.757+0000 7f46bfb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:31.758 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:31Z"} 2024-03-04T16:45:31.759 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:31Z"} 2024-03-04T16:45:31.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:31.759 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:31.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:31.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T16:45:36.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:36.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:36.761 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:36.855 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:36.873+0000 7ff002b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:36.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:36.874+0000 7ff002b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:36.875 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:36Z"} 2024-03-04T16:45:36.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:36Z"} 2024-03-04T16:45:36.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:36.876 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:36.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:36.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T16:45:41.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:41.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:41.877 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:41.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:41.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:41.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:42.017+0000 7fc380a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:41.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:42.018+0000 7fc380a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:42.021 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:42Z"} 2024-03-04T16:45:42.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:42Z"} 2024-03-04T16:45:42.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:42.023 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:42.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:42.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.662426_417.1 2024-03-04T16:45:47.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:47.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:47.024 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:47.071 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:47.071 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:47.113 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:47.132+0000 7f9ed1c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:47.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:47.132+0000 7f9ed1c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:47.133 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:47Z"} 2024-03-04T16:45:47.134 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:47Z"} 2024-03-04T16:45:47.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:47.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:47.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:47.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:45:52.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:52.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:52.135 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:52.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:52.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:52.233 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:52.251+0000 7fd0de131800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:52.234 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:52.252+0000 7fd0de131800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:52.254 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:52Z"} 2024-03-04T16:45:52.254 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:52Z"} 2024-03-04T16:45:52.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:52.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:52.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:52.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T16:45:57.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:57.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:45:57.256 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:45:57.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:45:57.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:45:57.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:57.435+0000 7f9b96814800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:57.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:45:57.436+0000 7f9b96814800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:45:57.438 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:57Z"} 2024-03-04T16:45:57.438 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:57Z"} 2024-03-04T16:45:57.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:45:57.439 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:57.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:45:57.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.644836_414.1 2024-03-04T16:46:02.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:02.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:02.441 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:02.493 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:02.493 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:02.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:02.557+0000 7f73784c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:02.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:02.558+0000 7f73784c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:02.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:02Z"} 2024-03-04T16:46:02.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:02Z"} 2024-03-04T16:46:02.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:02.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:02.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:02.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T16:46:07.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:07.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:07.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:07.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:07.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:07.654 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:07.672+0000 7fe1cd824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:07.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:07.673+0000 7fe1cd824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:07.674 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:07Z"} 2024-03-04T16:46:07.674 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:07Z"} 2024-03-04T16:46:07.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:07.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:07.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:07.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T16:46:12.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:12.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:12.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:12.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:12.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:12.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:12.796+0000 7f030ae12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:12.778 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:12.797+0000 7f030ae12800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:12.800 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:12Z"} 2024-03-04T16:46:12.800 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:12Z"} 2024-03-04T16:46:12.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:12.800 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:12.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:12.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T16:46:17.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:17.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:17.802 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:17.850 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:17.851 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:17.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:17.913+0000 7f4cd5824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:17.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:17.914+0000 7f4cd5824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:17.915 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:17Z"} 2024-03-04T16:46:17.915 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:17Z"} 2024-03-04T16:46:17.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:17.915 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:17.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:17.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:46:22.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:22.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:22.917 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:22.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:22.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:23.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:23.037+0000 7f1665311800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:23.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:23.038+0000 7f1665311800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:23.040 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:23Z"} 2024-03-04T16:46:23.040 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:23Z"} 2024-03-04T16:46:23.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:23.040 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:23.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:23.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T16:46:28.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:28.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:28.042 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:28.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:28.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:28.145 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:28.163+0000 7fd27fb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:28.146 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:28.164+0000 7fd27fb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:28.166 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:28Z"} 2024-03-04T16:46:28.166 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:28Z"} 2024-03-04T16:46:28.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:28.167 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:28.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:28.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T16:46:33.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:33.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:33.169 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:33.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:33.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:33.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:33.288+0000 7f9d6ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:33.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:33.289+0000 7f9d6ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:33.291 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:33Z"} 2024-03-04T16:46:33.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:33Z"} 2024-03-04T16:46:33.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:33.292 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:33.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:33.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T16:46:38.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:38.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:38.294 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:38.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:38.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:38.485 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:38.504+0000 7ff8fdd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:38.486 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:38.505+0000 7ff8fdd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:38.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:38Z"} 2024-03-04T16:46:38.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:38Z"} 2024-03-04T16:46:38.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:38.506 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:38.506 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:38.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T16:46:43.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:43.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:43.508 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:43.593 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:43.593 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:43.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:43.681+0000 7fae6a02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:43.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:43.682+0000 7fae6a02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:43.685 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:43Z"} 2024-03-04T16:46:43.685 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:43Z"} 2024-03-04T16:46:43.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:43.686 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:43.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:43.686 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T16:46:48.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:48.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:48.688 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:48.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:48.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:48.779 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:48.798+0000 7f86ce22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:48.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:48.799+0000 7f86ce22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:48.800 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:48Z"} 2024-03-04T16:46:48.800 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:48Z"} 2024-03-04T16:46:48.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:48.800 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:48.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:48.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T16:46:53.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:53.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:53.802 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:53.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:53.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:53.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:53.929+0000 7f7b70230800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:53.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:53.930+0000 7f7b70230800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:53.933 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:53Z"} 2024-03-04T16:46:53.933 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:53Z"} 2024-03-04T16:46:53.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:53.933 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:53.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:53.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.620918_410.1 2024-03-04T16:46:58.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:58.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:46:58.934 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:46:58.981 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:46:58.981 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:46:59.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:59.043+0000 7f4573b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:59.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:46:59.044+0000 7f4573b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:46:59.045 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:59Z"} 2024-03-04T16:46:59.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:59Z"} 2024-03-04T16:46:59.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:46:59.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:59.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:46:59.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T16:47:04.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:04.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:04.047 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:04.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:04.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:04.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:04.155+0000 7f2bd6a8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:04.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:04.156+0000 7f2bd6a8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:04.157 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:04Z"} 2024-03-04T16:47:04.157 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:04Z"} 2024-03-04T16:47:04.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:04.157 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:04.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:04.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T16:47:09.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:09.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:09.159 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:09.209 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:09.209 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:09.285+0000 7f13a8b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:09.285+0000 7f13a8b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:09.286 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:09Z"} 2024-03-04T16:47:09.286 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:09Z"} 2024-03-04T16:47:09.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:09.287 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:09.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:09.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T16:47:14.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:14.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:14.288 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:14.337 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:14.410+0000 7f420b737800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:14.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:14.411+0000 7f420b737800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:14.412 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:14Z"} 2024-03-04T16:47:14.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:14Z"} 2024-03-04T16:47:14.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:14.413 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:14.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:14.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T16:47:19.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:19.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:19.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:19.467 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:19.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:19.589+0000 7f99bc067800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:19.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:19.590+0000 7f99bc067800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:19.602 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:19Z"} 2024-03-04T16:47:19.602 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:19Z"} 2024-03-04T16:47:19.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:19.603 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:19.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:19.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T16:47:24.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:24.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:24.604 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:24.657 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:24.658 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:24.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:24.783+0000 7f6598c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:24.775 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:24.794+0000 7f6598c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:24.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:24Z"} 2024-03-04T16:47:24.848 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:24Z"} 2024-03-04T16:47:24.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:24.848 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:24.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:24.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T16:47:29.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:29.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:29.849 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:29.963+0000 7f20cd124800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:29.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:29.964+0000 7f20cd124800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:29.965 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:29Z"} 2024-03-04T16:47:29.965 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:29Z"} 2024-03-04T16:47:29.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:29.966 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:29.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:29.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:47:34.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:34.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:34.968 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:35.014 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:35.015 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:35.067 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:35.086+0000 7f284ed31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:35.067 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:35.087+0000 7f284ed31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:35.097 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:35Z"} 2024-03-04T16:47:35.097 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:35Z"} 2024-03-04T16:47:35.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:35.098 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:35.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:35.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.795745_422.1 2024-03-04T16:47:40.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:40.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:40.099 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:40.145 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:40.145 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:40.212+0000 7f05b5832800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:40.213+0000 7f05b5832800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:40.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:40Z"} 2024-03-04T16:47:40.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:40Z"} 2024-03-04T16:47:40.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:40.214 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:40.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:40.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T16:47:45.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:45.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:45.216 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:45.263 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:45.263 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:45.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:45.328+0000 7f82fd08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:45.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:45.329+0000 7f82fd08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:45.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:45Z"} 2024-03-04T16:47:45.330 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:45Z"} 2024-03-04T16:47:45.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:45.330 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:45.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:45.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T16:47:50.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:50.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:50.332 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:50.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:50.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:50.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:50.453+0000 7f222d93b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:50.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:50.454+0000 7f222d93b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:50.458 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:50Z"} 2024-03-04T16:47:50.458 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:50Z"} 2024-03-04T16:47:50.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:50.459 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:50.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:50.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T16:47:55.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:55.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:47:55.460 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:47:55.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:47:55.510 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:47:55.577 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:55.596+0000 7f96b8ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:55.578 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:47:55.597+0000 7f96b8ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:47:55.606 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:55Z"} 2024-03-04T16:47:55.606 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:55Z"} 2024-03-04T16:47:55.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:47:55.606 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:55.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:47:55.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:48:00.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:00.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:00.608 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:00.657 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:00.657 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:00.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:00.726+0000 7f51cac2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:00.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:00.727+0000 7f51cac2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:00.740 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:00Z"} 2024-03-04T16:48:00.740 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:00Z"} 2024-03-04T16:48:00.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:00.740 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:00.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:00.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.736351_420.1 2024-03-04T16:48:05.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:05.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:05.743 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:05.796 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:05.797 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:05.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:05.870+0000 7f0aee40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:05.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:05.871+0000 7f0aee40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:05.872 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:05Z"} 2024-03-04T16:48:05.872 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:05Z"} 2024-03-04T16:48:05.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:05.872 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:05.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:05.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T16:48:10.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:10.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:10.874 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:10.928 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:10.928 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:10.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:11.005+0000 7f6ea8c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:10.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:11.006+0000 7f6ea8c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:11.037 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:11Z"} 2024-03-04T16:48:11.037 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:11Z"} 2024-03-04T16:48:11.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:11.037 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:11.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:11.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T16:48:16.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:16.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:16.039 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:16.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:16.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:16.198 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:16.217+0000 7f09b5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:16.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:16.218+0000 7f09b5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:16.236 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:16Z"} 2024-03-04T16:48:16.236 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:16Z"} 2024-03-04T16:48:16.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:16.236 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:16.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:16.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.813215_426.1 2024-03-04T16:48:21.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:21.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:21.238 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:21.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:21.353+0000 7f3bbfe8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:21.334 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:21.354+0000 7f3bbfe8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:21Z"} 2024-03-04T16:48:21.360 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:21Z"} 2024-03-04T16:48:21.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:21.360 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:21.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:21.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T16:48:26.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:26.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:26.362 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:26.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:26.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:26.474+0000 7f9ec1111800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:26.475+0000 7f9ec1111800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:26.476 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:26Z"} 2024-03-04T16:48:26.477 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:26Z"} 2024-03-04T16:48:26.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:26.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:26.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:26.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.620918_410.1 2024-03-04T16:48:31.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:31.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:31.479 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:31.533 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:31.534 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:31.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:31.606+0000 7f5c35015800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:31.587 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:31.607+0000 7f5c35015800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:31.616 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:31Z"} 2024-03-04T16:48:31.616 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:31Z"} 2024-03-04T16:48:31.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:31.617 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:31.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:31.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T16:48:36.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:36.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:36.618 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:36.668 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:36.668 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:36.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:36.748+0000 7f9474742800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:36.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:36.749+0000 7f9474742800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:36.765 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:36Z"} 2024-03-04T16:48:36.765 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:36Z"} 2024-03-04T16:48:36.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.798294_423.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.798294Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:36.765 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:36.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.798294_423.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:36.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.798294_423.1 2024-03-04T16:48:41.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:41.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:41.767 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:41.873 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:41.893+0000 7fcff49af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:41.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:41.893+0000 7fcff49af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:41.899 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:41Z"} 2024-03-04T16:48:41.899 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:41Z"} 2024-03-04T16:48:41.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:41.900 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:41.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:41.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T16:48:46.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:46.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:46.901 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:46.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:46.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:47.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:47.036+0000 7fc9cdf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:47.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:47.037+0000 7fc9cdf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:47.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:47Z"} 2024-03-04T16:48:47.039 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:47Z"} 2024-03-04T16:48:47.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:47.040 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:47.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:47.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T16:48:52.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:52.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:52.042 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:52.091 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:52.091 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:52.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:52.171+0000 7f26f7324800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:52.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:52.172+0000 7f26f7324800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:52.198 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:52Z"} 2024-03-04T16:48:52.198 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:52Z"} 2024-03-04T16:48:52.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:52.198 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:52.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:52.198 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T16:48:57.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:57.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:48:57.200 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:48:57.281 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:48:57.281 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:48:57.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:57.361+0000 7f68d4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:57.342 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:48:57.362+0000 7f68d4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:48:57.363 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:57Z"} 2024-03-04T16:48:57.363 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:57Z"} 2024-03-04T16:48:57.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:48:57.363 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:57.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:48:57.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T16:49:02.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:02.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:02.365 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:02.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:02.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:02.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:02.482+0000 7f0f5a13b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:02.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:02.483+0000 7f0f5a13b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:02.482 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:02Z"} 2024-03-04T16:49:02.482 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:02Z"} 2024-03-04T16:49:02.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:02.483 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:02.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:02.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T16:49:07.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:07.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:07.485 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:07.530 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:07.530 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:07.577 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:07.597+0000 7fba32122800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:07.578 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:07.598+0000 7fba32122800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:07.600 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:07Z"} 2024-03-04T16:49:07.600 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:07Z"} 2024-03-04T16:49:07.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:07.600 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:07.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:07.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T16:49:12.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:12.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:12.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:12.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:12.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:12.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:12.722+0000 7f22cf7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:12.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:12.723+0000 7f22cf7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:12.730 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:12Z"} 2024-03-04T16:49:12.730 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:12Z"} 2024-03-04T16:49:12.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:12.730 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:12.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:12.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T16:49:17.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:17.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:17.731 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:17.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:17.779 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:17.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:17.859+0000 7ff9cb830800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:17.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:17.860+0000 7ff9cb830800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:17.870 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:17Z"} 2024-03-04T16:49:17.870 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:17Z"} 2024-03-04T16:49:17.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:17.871 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:17.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:17.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.644836_414.1 2024-03-04T16:49:22.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:22.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:22.872 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:22.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:22.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:22.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:22.993+0000 7f92eed11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:22.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:22.994+0000 7f92eed11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:23.000 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:22Z"} 2024-03-04T16:49:23.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:22Z"} 2024-03-04T16:49:23.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:23.001 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:23.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:23.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T16:49:28.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:28.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:28.003 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:28.053 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:28.053 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:28.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:28.123+0000 7f12b993b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:28.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:28.124+0000 7f12b993b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:28.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:28Z"} 2024-03-04T16:49:28.125 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:28Z"} 2024-03-04T16:49:28.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:28.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:28.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:28.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T16:49:33.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:33.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:33.128 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:33.177 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:33.178 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:33.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:33.261+0000 7fecf0b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:33.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:33.262+0000 7fecf0b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:33.262 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:33Z"} 2024-03-04T16:49:33.263 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:33Z"} 2024-03-04T16:49:33.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:33.263 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:33.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:33.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T16:49:38.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:38.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:38.265 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:38.319 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:38.320 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:38.395 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:38.415+0000 7fc8f7fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:38.416+0000 7fc8f7fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:38.427 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:38Z"} 2024-03-04T16:49:38.428 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:38Z"} 2024-03-04T16:49:38.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:38.428 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:38.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:38.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T16:49:43.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:43.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:43.430 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:43.480 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:43.481 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:43.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:43.561+0000 7f41e7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:43.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:43.562+0000 7f41e7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:43.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:43Z"} 2024-03-04T16:49:43.567 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:43Z"} 2024-03-04T16:49:43.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:43.568 DEBUG:rgw_multi.tests:log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:43.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:43.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.799310_424.1 2024-03-04T16:49:48.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2024-03-04T16:49:48.572 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-45 2024-03-04T16:49:48.573 DEBUG:boto:path=/sbnvju-45/ 2024-03-04T16:49:48.573 DEBUG:boto:auth_path=/sbnvju-45/ 2024-03-04T16:49:48.573 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.573 DEBUG:boto:Path: /sbnvju-45/ 2024-03-04T16:49:48.573 DEBUG:boto:Data: 2024-03-04T16:49:48.573 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.573 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:49:48.574 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.574 DEBUG:boto:Params: {} 2024-03-04T16:49:48.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:49:48.575 DEBUG:boto:Token: None 2024-03-04T16:49:48.575 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-45/ 2024-03-04T16:49:48.575 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ey3kEGarYfskZiPhWj/WUjpZqnY= 2024-03-04T16:49:48.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ey3kEGarYfskZiPhWj/WUjpZqnY=', 'Content-Length': '0'} 2024-03-04T16:49:48.599 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a06a1596a7b8f2bc-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.599 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-46 2024-03-04T16:49:48.599 DEBUG:boto:path=/sbnvju-46/ 2024-03-04T16:49:48.599 DEBUG:boto:auth_path=/sbnvju-46/ 2024-03-04T16:49:48.599 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.600 DEBUG:boto:Path: /sbnvju-46/ 2024-03-04T16:49:48.600 DEBUG:boto:Data: 2024-03-04T16:49:48.600 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:49:48.600 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.600 DEBUG:boto:Params: {} 2024-03-04T16:49:48.600 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:49:48.601 DEBUG:boto:Token: None 2024-03-04T16:49:48.601 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-46/ 2024-03-04T16:49:48.601 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:J/uSG4lXjAFEkNWf2LScx0M+pR0= 2024-03-04T16:49:48.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J/uSG4lXjAFEkNWf2LScx0M+pR0=', 'Content-Length': '0'} 2024-03-04T16:49:48.642 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b5b37f0d6004570-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.642 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-47 2024-03-04T16:49:48.642 DEBUG:boto:path=/sbnvju-47/ 2024-03-04T16:49:48.642 DEBUG:boto:auth_path=/sbnvju-47/ 2024-03-04T16:49:48.642 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.642 DEBUG:boto:Path: /sbnvju-47/ 2024-03-04T16:49:48.642 DEBUG:boto:Data: 2024-03-04T16:49:48.642 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.643 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:49:48.643 DEBUG:boto:Port: 8001 2024-03-04T16:49:48.643 DEBUG:boto:Params: {} 2024-03-04T16:49:48.643 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T16:49:48.643 DEBUG:boto:Token: None 2024-03-04T16:49:48.643 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-47/ 2024-03-04T16:49:48.643 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BANL2whvC2cooIw8JdPAINxgQMY= 2024-03-04T16:49:48.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BANL2whvC2cooIw8JdPAINxgQMY=', 'Content-Length': '0'} 2024-03-04T16:49:48.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f2adb32108ea621-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.683 DEBUG:boto:path=/sbnvju-45/ 2024-03-04T16:49:48.683 DEBUG:boto:auth_path=/sbnvju-45/ 2024-03-04T16:49:48.683 DEBUG:boto:Method: HEAD 2024-03-04T16:49:48.683 DEBUG:boto:Path: /sbnvju-45/ 2024-03-04T16:49:48.683 DEBUG:boto:Data: 2024-03-04T16:49:48.684 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.684 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:49:48.684 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.684 DEBUG:boto:Params: {} 2024-03-04T16:49:48.684 DEBUG:boto:Token: None 2024-03-04T16:49:48.684 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-45/ 2024-03-04T16:49:48.684 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:p62tCX6TSDrdpSvvdvg7wmsSDWE= 2024-03-04T16:49:48.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p62tCX6TSDrdpSvvdvg7wmsSDWE=', 'Content-Length': '0'} 2024-03-04T16:49:48.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d034b72e28fce359-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.687 DEBUG:boto:path=/sbnvju-45/key 2024-03-04T16:49:48.687 DEBUG:boto:auth_path=/sbnvju-45/key 2024-03-04T16:49:48.687 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.687 DEBUG:boto:Path: /sbnvju-45/key 2024-03-04T16:49:48.687 DEBUG:boto:Data: 2024-03-04T16:49:48.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T16:49:48.687 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:49:48.687 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.688 DEBUG:boto:Params: {} 2024-03-04T16:49:48.688 DEBUG:boto:Token: None 2024-03-04T16:49:48.688 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-45/key 2024-03-04T16:49:48.688 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:BErwpOVkRokOHuRI//Ullm2brEI= 2024-03-04T16:49:48.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BErwpOVkRokOHuRI//Ullm2brEI='} 2024-03-04T16:49:48.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009864630a6bb56864-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.699 DEBUG:boto:path=/sbnvju-46/ 2024-03-04T16:49:48.699 DEBUG:boto:auth_path=/sbnvju-46/ 2024-03-04T16:49:48.699 DEBUG:boto:Method: HEAD 2024-03-04T16:49:48.699 DEBUG:boto:Path: /sbnvju-46/ 2024-03-04T16:49:48.699 DEBUG:boto:Data: 2024-03-04T16:49:48.699 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.699 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:49:48.699 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.699 DEBUG:boto:Params: {} 2024-03-04T16:49:48.699 DEBUG:boto:Token: None 2024-03-04T16:49:48.700 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-46/ 2024-03-04T16:49:48.700 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:36K4jKrqSho7/C16LolxuKdMrDQ= 2024-03-04T16:49:48.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:36K4jKrqSho7/C16LolxuKdMrDQ=', 'Content-Length': '0'} 2024-03-04T16:49:48.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064de92858d525acc-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.705 DEBUG:boto:path=/sbnvju-46/key 2024-03-04T16:49:48.705 DEBUG:boto:auth_path=/sbnvju-46/key 2024-03-04T16:49:48.705 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.705 DEBUG:boto:Path: /sbnvju-46/key 2024-03-04T16:49:48.705 DEBUG:boto:Data: 2024-03-04T16:49:48.705 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T16:49:48.705 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T16:49:48.705 DEBUG:boto:Port: 8000 2024-03-04T16:49:48.706 DEBUG:boto:Params: {} 2024-03-04T16:49:48.706 DEBUG:boto:Token: None 2024-03-04T16:49:48.706 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-46/key 2024-03-04T16:49:48.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:M51kz33om5g8cjSMuZ64Yo8xGB0= 2024-03-04T16:49:48.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M51kz33om5g8cjSMuZ64Yo8xGB0='} 2024-03-04T16:49:48.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003754ca8ce2296fcf-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.754 DEBUG:boto:path=/sbnvju-47/ 2024-03-04T16:49:48.754 DEBUG:boto:auth_path=/sbnvju-47/ 2024-03-04T16:49:48.754 DEBUG:boto:Method: HEAD 2024-03-04T16:49:48.754 DEBUG:boto:Path: /sbnvju-47/ 2024-03-04T16:49:48.754 DEBUG:boto:Data: 2024-03-04T16:49:48.754 DEBUG:boto:Headers: {} 2024-03-04T16:49:48.754 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:49:48.754 DEBUG:boto:Port: 8001 2024-03-04T16:49:48.754 DEBUG:boto:Params: {} 2024-03-04T16:49:48.755 DEBUG:boto:Token: None 2024-03-04T16:49:48.755 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-47/ 2024-03-04T16:49:48.755 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:S3ikBNw0UEFs9U2RZxOmPFPpF+w= 2024-03-04T16:49:48.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S3ikBNw0UEFs9U2RZxOmPFPpF+w=', 'Content-Length': '0'} 2024-03-04T16:49:48.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b29d12d1820b6975-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.758 DEBUG:boto:path=/sbnvju-47/key 2024-03-04T16:49:48.758 DEBUG:boto:auth_path=/sbnvju-47/key 2024-03-04T16:49:48.758 DEBUG:boto:Method: PUT 2024-03-04T16:49:48.758 DEBUG:boto:Path: /sbnvju-47/key 2024-03-04T16:49:48.758 DEBUG:boto:Data: 2024-03-04T16:49:48.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T16:49:48.758 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T16:49:48.758 DEBUG:boto:Port: 8001 2024-03-04T16:49:48.758 DEBUG:boto:Params: {} 2024-03-04T16:49:48.759 DEBUG:boto:Token: None 2024-03-04T16:49:48.759 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 04 Mar 2024 16:49:48 GMT /sbnvju-47/key 2024-03-04T16:49:48.759 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KFDMWCYkP+ZtFbEAq7voJBNay0E= 2024-03-04T16:49:48.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KFDMWCYkP+ZtFbEAq7voJBNay0E='} 2024-03-04T16:49:48.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050ccbb2b1ef674e7-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:49:48.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:48.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:48.766 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:49:48.890 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709570988.692350_570.1","last_update":"2024-03-04T16:49:48.692350Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:49:48.891 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:48.891 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:49:48.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:48.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:48.891 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:48.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:48.941 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:48.995 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:49.015+0000 7fd334d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:48.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:49.016+0000 7fd334d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:49.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:49Z"} 2024-03-04T16:49:49.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:49Z"} 2024-03-04T16:49:49.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:49.033 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:49.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:49.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T16:49:54.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:54.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:54.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:54.080 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:54.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:54.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:54.157+0000 7f916b610800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:54.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:54.158+0000 7f916b610800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:54.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:54Z"} 2024-03-04T16:49:54.165 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:54Z"} 2024-03-04T16:49:54.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:54.166 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:54.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:54.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T16:49:59.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:59.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:49:59.168 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:49:59.228 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:49:59.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:49:59.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:59.303+0000 7fb6e3639800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:59.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:49:59.304+0000 7fb6e3639800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:49:59.305 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:59Z"} 2024-03-04T16:49:59.305 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:59Z"} 2024-03-04T16:49:59.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:49:59.306 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:59.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:49:59.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T16:50:04.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:04.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:04.307 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:04.353 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:04.354 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:04.410 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:04.430+0000 7f4476b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:04.410 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:04.431+0000 7f4476b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:04.431 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:04Z"} 2024-03-04T16:50:04.431 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:04Z"} 2024-03-04T16:50:04.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:04.431 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:04.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:04.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.641235_412.1 2024-03-04T16:50:09.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:09.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:09.433 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:09.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:09.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:09.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:09.548+0000 7ff4c8a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:09.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:09.549+0000 7ff4c8a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:09.552 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:09Z"} 2024-03-04T16:50:09.552 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:09Z"} 2024-03-04T16:50:09.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:09.553 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:09.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:09.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T16:50:14.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:14.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:14.555 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:14.603 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:14.603 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:14.652 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:14.672+0000 7f6a67022800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:14.652 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:14.673+0000 7f6a67022800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:14.675 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:14Z"} 2024-03-04T16:50:14.675 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:14Z"} 2024-03-04T16:50:14.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:14.676 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:14.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:14.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T16:50:19.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:19.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:19.678 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:19.728 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:19.728 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:19.812 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:19.832+0000 7f48a1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:19.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:19.833+0000 7f48a1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:19.837 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:19Z"} 2024-03-04T16:50:19.837 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:19Z"} 2024-03-04T16:50:19.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:19.837 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:19.837 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:19.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:50:24.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:24.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:24.839 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:24.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:24.964+0000 7f1ca8b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:24.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:24.965+0000 7f1ca8b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:24.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:24Z"} 2024-03-04T16:50:24.978 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:24Z"} 2024-03-04T16:50:24.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:24.978 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:24.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:24.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T16:50:29.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:29.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:29.980 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:30.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:30.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:30.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:30.105+0000 7fa6e7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:30.087 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:30.108+0000 7fa6e7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:30.172 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:30Z"} 2024-03-04T16:50:30.172 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:30Z"} 2024-03-04T16:50:30.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:30.172 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:30.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:30.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T16:50:35.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:35.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:35.174 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:35.227 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:35.227 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:35.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:35.309+0000 7fdd78636800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:35.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:35.310+0000 7fdd78636800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:35.311 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:35Z"} 2024-03-04T16:50:35.311 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:35Z"} 2024-03-04T16:50:35.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:35.312 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:35.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:35.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.641235_412.1 2024-03-04T16:50:40.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:40.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:40.313 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:40.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:40.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:40.441 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:40.462+0000 7f25df616800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:40.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:40.463+0000 7f25df616800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:40.478 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:40Z"} 2024-03-04T16:50:40.478 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:40Z"} 2024-03-04T16:50:40.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:40.479 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:40.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:40.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:50:45.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:45.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:45.481 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:45.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:45.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:45.585 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:45.605+0000 7f2b7d067800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:45.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:45.606+0000 7f2b7d067800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:45.610 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:45Z"} 2024-03-04T16:50:45.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:45Z"} 2024-03-04T16:50:45.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:45.611 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:45.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:45.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.738966_421.1 2024-03-04T16:50:50.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:50.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:50.613 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:50.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:50.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:50.706 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:50.727+0000 7f82e482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:50.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:50.728+0000 7f82e482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:50.739 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:50Z"} 2024-03-04T16:50:50.739 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:50Z"} 2024-03-04T16:50:50.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:50.740 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:50.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:50.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.813215_426.1 2024-03-04T16:50:55.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:55.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:50:55.742 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:50:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:50:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:50:55.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:55.878+0000 7fc02e288800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:55.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:50:55.879+0000 7fc02e288800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:50:55.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:55Z"} 2024-03-04T16:50:55.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:55Z"} 2024-03-04T16:50:55.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:50:55.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:55.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:50:55.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:51:00.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:00.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:00.880 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:00.945 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:00.945 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:01.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:01.032+0000 7feb11b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:01.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:01.032+0000 7feb11b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:01.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:01Z"} 2024-03-04T16:51:01.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:01Z"} 2024-03-04T16:51:01.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:01.033 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:01.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:01.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:51:06.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:06.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:06.035 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:06.088 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:06.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:06.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:06.163+0000 7fd1fc1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:06.143 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:06.164+0000 7fd1fc1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:06.167 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:06Z"} 2024-03-04T16:51:06.167 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:06Z"} 2024-03-04T16:51:06.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:06.168 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:06.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:06.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:51:11.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:11.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:11.170 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:11.224 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:11.224 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:11.297+0000 7f4841e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:11.298+0000 7f4841e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:11.298 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:11Z"} 2024-03-04T16:51:11.299 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:11Z"} 2024-03-04T16:51:11.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:11.299 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:11.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:11.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T16:51:16.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:16.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:16.301 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:16.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:16.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:16.475 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:16.496+0000 7f7143824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:16.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:16.498+0000 7f7143824800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:16.498 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:16Z"} 2024-03-04T16:51:16.499 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:16Z"} 2024-03-04T16:51:16.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:16.499 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:16.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:16.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:51:21.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:21.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:21.500 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:21.581 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:21.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:21.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:21.705+0000 7f98fe733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:21.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:21.710+0000 7f98fe733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:21.714 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:21Z"} 2024-03-04T16:51:21.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:21Z"} 2024-03-04T16:51:21.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:21.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:21.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:21.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T16:51:26.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:26.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:26.716 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:26.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:26.859+0000 7fdccc424800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:26.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:26.860+0000 7fdccc424800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:26.862 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:26Z"} 2024-03-04T16:51:26.863 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:26Z"} 2024-03-04T16:51:26.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:26.863 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:26.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:26.863 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.801670_425.1 2024-03-04T16:51:31.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:31.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:31.864 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:31.911 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:31.911 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:31.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:31.992+0000 7f163973b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:31.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:31.993+0000 7f163973b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:32.002 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:31Z"} 2024-03-04T16:51:32.003 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:31Z"} 2024-03-04T16:51:32.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:32.003 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:32.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:32.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T16:51:37.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:37.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:37.005 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:37.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:37.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:37.109 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:37.130+0000 7f046c01b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:37.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:37.131+0000 7f046c01b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:37.132 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:37Z"} 2024-03-04T16:51:37.132 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:37Z"} 2024-03-04T16:51:37.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.798294_423.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.798294Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:37.132 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:37.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.798294_423.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:37.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.798294_423.1 2024-03-04T16:51:42.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:42.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:42.134 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:42.180 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:42.180 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:42.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:42.248+0000 7f6e2428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:42.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:42.249+0000 7f6e2428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:42.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:42Z"} 2024-03-04T16:51:42.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:42Z"} 2024-03-04T16:51:42.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:42.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:42.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:42.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:51:47.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:47.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:47.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:47.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:47.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:47.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:47.368+0000 7f622e631800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:47.348 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:47.369+0000 7f622e631800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:47.372 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:47Z"} 2024-03-04T16:51:47.372 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:47Z"} 2024-03-04T16:51:47.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:47.373 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:47.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:47.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.841505_432.1 2024-03-04T16:51:52.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:52.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:52.374 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:52.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:52.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:52.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:52.488+0000 7fc0f5d21800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:52.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:52.489+0000 7fc0f5d21800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:52.490 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:52Z"} 2024-03-04T16:51:52.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:52Z"} 2024-03-04T16:51:52.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:52.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:52.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:52.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:51:57.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:57.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:51:57.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:51:57.539 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:51:57.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:51:57.589 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:57.610+0000 7fcc83a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:57.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:51:57.611+0000 7fcc83a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:51:57.611 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:57Z"} 2024-03-04T16:51:57.611 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:57Z"} 2024-03-04T16:51:57.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:51:57.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:57.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:51:57.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T16:52:02.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:02.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:02.614 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:02.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:02.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:02.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:02.730+0000 7fa773013800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:02.710 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:02.731+0000 7fa773013800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:02.731 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:02Z"} 2024-03-04T16:52:02.731 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:02Z"} 2024-03-04T16:52:02.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:02.732 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:02.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:02.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.838183_430.1 2024-03-04T16:52:07.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:07.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:07.733 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:07.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:07.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:07.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:07.864+0000 7f04dd131800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:07.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:07.865+0000 7f04dd131800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:07.865 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:07Z"} 2024-03-04T16:52:07.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:07Z"} 2024-03-04T16:52:07.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:07.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:07.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:07.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T16:52:12.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:12.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:12.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:12.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:12.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:12.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:13.008+0000 7fc235613800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:12.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:13.009+0000 7fc235613800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:13.027 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:13Z"} 2024-03-04T16:52:13.027 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:13Z"} 2024-03-04T16:52:13.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:13.027 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:13.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:13.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.969696_444.1 2024-03-04T16:52:18.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:18.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:18.030 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:18.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:18.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:18.129 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:18.150+0000 7f9005d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:18.130 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:18.151+0000 7f9005d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:18.152 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:18Z"} 2024-03-04T16:52:18.152 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:18Z"} 2024-03-04T16:52:18.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:18.152 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:18.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:18.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:52:23.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:23.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:23.154 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:23.236 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:23.236 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:23.336 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:23.357+0000 7fced6331800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:23.337 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:23.358+0000 7fced6331800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:23.374 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:23Z"} 2024-03-04T16:52:23.374 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:23Z"} 2024-03-04T16:52:23.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:23.374 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:23.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:23.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T16:52:28.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:28.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:28.376 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:28.440 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:28.440 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:28.504 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:28.525+0000 7fe103930800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:28.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:28.526+0000 7fe103930800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:28.530 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:28Z"} 2024-03-04T16:52:28.530 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:28Z"} 2024-03-04T16:52:28.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:28.531 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:28.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:28.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:52:33.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:33.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:33.533 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:33.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:33.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:33.636 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:33.657+0000 7f4c7df2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:33.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:33.658+0000 7f4c7df2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:33.661 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:33Z"} 2024-03-04T16:52:33.661 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:33Z"} 2024-03-04T16:52:33.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:33.662 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:33.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:33.662 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:52:38.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:38.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:38.663 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:38.710 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:38.710 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:38.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:38.794+0000 7eff6868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:38.773 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:38.795+0000 7eff6868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:38.798 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:38Z"} 2024-03-04T16:52:38.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:38Z"} 2024-03-04T16:52:38.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:38.798 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:38.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:38.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.840554_431.1 2024-03-04T16:52:43.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:43.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:43.799 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:43.846 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:43.846 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:43.906 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:43.927+0000 7f29bf010800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:43.907 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:43.928+0000 7f29bf010800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:43.934 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:43Z"} 2024-03-04T16:52:43.934 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:43Z"} 2024-03-04T16:52:43.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:43.935 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:43.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:43.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.736351_420.1 2024-03-04T16:52:48.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:48.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:48.936 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:48.983 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:48.983 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:49.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:49.064+0000 7f7cc173c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:49.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:49.065+0000 7f7cc173c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:49.066 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:49Z"} 2024-03-04T16:52:49.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:49Z"} 2024-03-04T16:52:49.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:49.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:49.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:49.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T16:52:54.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:54.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:54.069 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:54.115 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:54.115 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:54.163 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:54.184+0000 7fb599fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:54.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:54.185+0000 7fb599fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:54Z"} 2024-03-04T16:52:54.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:54Z"} 2024-03-04T16:52:54.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:54.186 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:54.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:54.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.816633_428.1 2024-03-04T16:52:59.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:59.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:52:59.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:52:59.234 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:52:59.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:52:59.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:59.311+0000 7f862ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:59.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:52:59.312+0000 7f862ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:52:59.311 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:59Z"} 2024-03-04T16:52:59.311 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:59Z"} 2024-03-04T16:52:59.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:52:59.312 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:59.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:52:59.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:53:04.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:04.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:04.313 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:04.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:04.531+0000 7f3338729800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:04.510 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:04.532+0000 7f3338729800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:04.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:04Z"} 2024-03-04T16:53:04.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:04Z"} 2024-03-04T16:53:04.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:04.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:04.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:04.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T16:53:09.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:09.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:09.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:09.595 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:09.595 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:09.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:09.668+0000 7eff4c616800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:09.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:09.669+0000 7eff4c616800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:09.675 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:09Z"} 2024-03-04T16:53:09.675 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:09Z"} 2024-03-04T16:53:09.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:09.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:09.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:09.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T16:53:14.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:14.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:14.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:14.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:14.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:14.771 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:14.793+0000 7f479cc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:14.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:14.794+0000 7f479cc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:14.793 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:14Z"} 2024-03-04T16:53:14.793 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:14Z"} 2024-03-04T16:53:14.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:14.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:14.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:14.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:53:19.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:19.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:19.795 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:19.843 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:19.843 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:19.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:19.932+0000 7f56dde11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:19.912 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:19.933+0000 7f56dde11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:19.942 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:19Z"} 2024-03-04T16:53:19.942 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:19Z"} 2024-03-04T16:53:19.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:19.942 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:19.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:19.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T16:53:24.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:24.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:24.944 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:24.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:24.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:25.048 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:25.070+0000 7f6009f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:25.049 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:25.071+0000 7f6009f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:25.070 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:25Z"} 2024-03-04T16:53:25.070 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:25Z"} 2024-03-04T16:53:25.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:25.070 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:25.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:25.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T16:53:30.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:30.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:30.072 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:30.118 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:30.119 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:30.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:30.199+0000 7f1218e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:30.178 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:30.200+0000 7f1218e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:30.205 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:30Z"} 2024-03-04T16:53:30.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:30Z"} 2024-03-04T16:53:30.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:30.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:30.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:30.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T16:53:35.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:35.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:35.208 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:35.255 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:35.255 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:35.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:35.335+0000 7f12a4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:35.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:35.336+0000 7f12a4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:35.344 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:35Z"} 2024-03-04T16:53:35.345 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:35Z"} 2024-03-04T16:53:35.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:35.345 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:35.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:35.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.801670_425.1 2024-03-04T16:53:40.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:40.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:40.347 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:40.397 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:40.397 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:40.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:40.475+0000 7f37ca7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:40.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:40.476+0000 7f37ca7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:40.476 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:40Z"} 2024-03-04T16:53:40.476 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:40Z"} 2024-03-04T16:53:40.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:40.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:40.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:40.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T16:53:45.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:45.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:45.478 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:45.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:45.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:45.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:45.625+0000 7fcfe0644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:45.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:45.626+0000 7fcfe0644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:45.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:45Z"} 2024-03-04T16:53:45.633 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:45Z"} 2024-03-04T16:53:45.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:45.633 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:45.633 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:45.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.927093_438.1 2024-03-04T16:53:50.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:50.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:50.635 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:50.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:50.761+0000 7f171fdaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:50.740 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:50.762+0000 7f171fdaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:50.767 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:50Z"} 2024-03-04T16:53:50.768 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:50Z"} 2024-03-04T16:53:50.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:50.768 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:50.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:50.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T16:53:55.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:55.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:53:55.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:53:55.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:53:55.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:53:55.871 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:55.893+0000 7fd82a644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:55.872 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:53:55.894+0000 7fd82a644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:53:55.900 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:55Z"} 2024-03-04T16:53:55.900 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:55Z"} 2024-03-04T16:53:55.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:53:55.901 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:55.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:53:55.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:54:00.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:00.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:00.902 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:00.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:00.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:01.000 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:01.022+0000 7f05acb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:01.001 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:01.023+0000 7f05acb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:01.022 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:01Z"} 2024-03-04T16:54:01.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:01Z"} 2024-03-04T16:54:01.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:01.023 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:01.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:01.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.736351_420.1 2024-03-04T16:54:06.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:06.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:06.024 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:06.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:06.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:06.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:06.143+0000 7fbf36444800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:06.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:06.144+0000 7fbf36444800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:06.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:06Z"} 2024-03-04T16:54:06.146 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:06Z"} 2024-03-04T16:54:06.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:06.146 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:06.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:06.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T16:54:11.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:11.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:11.149 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:11.202 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:11.202 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:11.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:11.276+0000 7fe0d2a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:11.255 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:11.277+0000 7fe0d2a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:11.278 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:11Z"} 2024-03-04T16:54:11.278 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:11Z"} 2024-03-04T16:54:11.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:11.278 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:11.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:11.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.602798_406.1 2024-03-04T16:54:16.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:16.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:16.279 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:16.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:16.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:16.418+0000 7f012b234800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:16.397 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:16.419+0000 7f012b234800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:16.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:16Z"} 2024-03-04T16:54:16.419 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:16Z"} 2024-03-04T16:54:16.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:16.419 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:16.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:16.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T16:54:21.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:21.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:21.421 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:21.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:21.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:21.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:21.559+0000 7f19f5ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:21.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:21.560+0000 7f19f5ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:21.562 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:21Z"} 2024-03-04T16:54:21.562 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:21Z"} 2024-03-04T16:54:21.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:21.563 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:21.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:21.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.602798_406.1 2024-03-04T16:54:26.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:26.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:26.565 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:26.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:26.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:26.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:26.709+0000 7fb106e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:26.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:26.710+0000 7fb106e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:26.717 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:26Z"} 2024-03-04T16:54:26.718 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:26Z"} 2024-03-04T16:54:26.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:26.718 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:26.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:26.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T16:54:31.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:31.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:31.720 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:31.774 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:31.774 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:31.919+0000 7fd89362d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:31.899 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:31.921+0000 7fd89362d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:31.969 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:31Z"} 2024-03-04T16:54:31.970 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:31Z"} 2024-03-04T16:54:31.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:31.970 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:31.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:31.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T16:54:36.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:36.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:36.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:37.053 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:37.054 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:37.171+0000 7f14daf30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:37.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:37.172+0000 7f14daf30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:37.173 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:37Z"} 2024-03-04T16:54:37.174 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:37Z"} 2024-03-04T16:54:37.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:37.174 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:37.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:37.174 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.643873_413.1 2024-03-04T16:54:42.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:42.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:42.175 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:42.221 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:42.221 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:42.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:42.305+0000 7fd486130800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:42.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:42.306+0000 7fd486130800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:42.304 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:42Z"} 2024-03-04T16:54:42.305 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:42Z"} 2024-03-04T16:54:42.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:42.305 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:42.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:42.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T16:54:47.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:47.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:47.308 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:47.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:47.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:47.428 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:47.450+0000 7fe4ae022800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:47.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:47.451+0000 7fe4ae022800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:47.451 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:47Z"} 2024-03-04T16:54:47.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:47Z"} 2024-03-04T16:54:47.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:47.452 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:47.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:47.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.643873_413.1 2024-03-04T16:54:52.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:52.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:54:52.454 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:54:52.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:52.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:52.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:52.576+0000 7fb8ec62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:52.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:54:52.577+0000 7fb8ec62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:54:52.576 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:52Z"} 2024-03-04T16:54:52.577 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:52Z"} 2024-03-04T16:54:52.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:54:52.577 DEBUG:rgw_multi.tests:log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:52.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:54:52.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.860903_435.1 2024-03-04T16:54:57.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-03-04T16:54:57.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:54:57.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:54:57.582 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:54:57.629 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:57.629 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:57.668 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T16:54:57.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:54:57.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T16:54:57.669 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T16:54:57.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:54:57.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:54:57.794 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone fc294220-5038-4f25-aeee-cac2c5838729 2024-03-04T16:54:57.840 INFO:teuthology.orchestra.run.smithi071.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":4,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","endpoints":["http://smithi027.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","val":210978822},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T16:54:57.841 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-03-04T16:55:57.842 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-48 2024-03-04T16:55:57.842 DEBUG:boto:path=/sbnvju-48/ 2024-03-04T16:55:57.843 DEBUG:boto:auth_path=/sbnvju-48/ 2024-03-04T16:55:57.843 DEBUG:boto:Method: PUT 2024-03-04T16:55:57.843 DEBUG:boto:Path: /sbnvju-48/ 2024-03-04T16:55:57.843 DEBUG:boto:Data: 2024-03-04T16:55:57.843 DEBUG:boto:Headers: {} 2024-03-04T16:55:57.843 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:55:57.843 DEBUG:boto:Port: 8000 2024-03-04T16:55:57.843 DEBUG:boto:Params: {} 2024-03-04T16:55:57.844 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T16:55:57.844 DEBUG:boto:Token: None 2024-03-04T16:55:57.845 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 16:55:57 GMT /sbnvju-48/ 2024-03-04T16:55:57.845 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iu4mpwfyx0WX+B1UwAZAUdiB3yk= 2024-03-04T16:55:57.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:55:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iu4mpwfyx0WX+B1UwAZAUdiB3yk=', 'Content-Length': '0'} 2024-03-04T16:55:57.869 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d215f6a777946ae6-0065e5fd1d-5285-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:55:57.870 DEBUG:boto:path=/sbnvju-48/testredirect 2024-03-04T16:55:57.870 DEBUG:boto:auth_path=/sbnvju-48/testredirect 2024-03-04T16:55:57.870 DEBUG:boto:Method: PUT 2024-03-04T16:55:57.870 DEBUG:boto:Path: /sbnvju-48/testredirect 2024-03-04T16:55:57.870 DEBUG:boto:Data: 2024-03-04T16:55:57.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T16:55:57.870 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T16:55:57.870 DEBUG:boto:Port: 8000 2024-03-04T16:55:57.870 DEBUG:boto:Params: {} 2024-03-04T16:55:57.870 DEBUG:boto:Token: None 2024-03-04T16:55:57.870 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 16:55:57 GMT /sbnvju-48/testredirect 2024-03-04T16:55:57.871 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Wt+hCQK+q9Elh3WThm0/KuXfGJU= 2024-03-04T16:55:57.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:55:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Wt+hCQK+q9Elh3WThm0/KuXfGJU='} 2024-03-04T16:55:57.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adaf49366724cc99-0065e5fd1d-5285-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T16:55:57.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:55:57.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:55:57.881 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:55:57.934 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:55:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:55:57.987 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T16:55:58.011 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709571357.890340_643.1","last_update":"2024-03-04T16:55:57.890340Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T16:55:58.011 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:55:58.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T16:55:58.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:55:58.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:55:58.012 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:55:58.077 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:55:58.077 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:55:58.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:55:58.161+0000 7f0d8c135800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:55:58.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:55:58.162+0000 7f0d8c135800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:55:58.162 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:55:58Z"} 2024-03-04T16:55:58.162 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:55:58Z"} 2024-03-04T16:55:58.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:55:58.162 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:55:58.162 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:55:58.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T16:56:03.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:03.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:03.164 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:03.290+0000 7f3fa693c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:03.291+0000 7f3fa693c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:03.296 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:03Z"} 2024-03-04T16:56:03.296 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:03Z"} 2024-03-04T16:56:03.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:03.296 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:03.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:03.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T16:56:08.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:08.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:08.298 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:08.344 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:08.345 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:08.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:08.414+0000 7fe692c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:08.393 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:08.415+0000 7fe692c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:08.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:08Z"} 2024-03-04T16:56:08.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:08Z"} 2024-03-04T16:56:08.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:08.414 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:08.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:08.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:56:13.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:13.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:13.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:13.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:13.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:13.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:13.550+0000 7f94b832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:13.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:13.551+0000 7f94b832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:13.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:13Z"} 2024-03-04T16:56:13.561 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:13Z"} 2024-03-04T16:56:13.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:13.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:13.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:13.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:56:18.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:18.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:18.563 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:18.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:18.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:18.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:18.686+0000 7f7b68929800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:18.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:18.687+0000 7f7b68929800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:18.686 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:18Z"} 2024-03-04T16:56:18.686 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:18Z"} 2024-03-04T16:56:18.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:18.687 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:18.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:18.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.858235_434.1 2024-03-04T16:56:23.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:23.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:23.688 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:23.741 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:23.741 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:23.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:23.821+0000 7fde9c224800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:23.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:23.822+0000 7fde9c224800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:23.821 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:23Z"} 2024-03-04T16:56:23.821 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:23Z"} 2024-03-04T16:56:23.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:23.822 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:23.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:23.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:56:28.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:28.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:28.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:28.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:28.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:28.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:28.980+0000 7fb21e831800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:28.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:28.982+0000 7fb21e831800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:29.060 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:28Z"} 2024-03-04T16:56:29.061 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:28Z"} 2024-03-04T16:56:29.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:29.061 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:29.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:29.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:56:29.198 INFO:tasks.rgw.c1.client.1.smithi027.stdout:2024-03-04T16:56:29.221+0000 7fc935f0d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T16:56:34.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:34.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:34.063 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:34.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:34.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:34.172 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:34.195+0000 7fd7ee328800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:34.173 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:34.195+0000 7fd7ee328800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:34.194 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:34Z"} 2024-03-04T16:56:34.194 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:34Z"} 2024-03-04T16:56:34.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:34.195 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:34.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:34.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T16:56:39.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:39.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:39.196 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:39.334+0000 7f88cc531800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:39.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:39.335+0000 7f88cc531800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:39.340 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:39Z"} 2024-03-04T16:56:39.341 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:39Z"} 2024-03-04T16:56:39.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:39.341 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:39.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:39.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T16:56:44.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:44.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:44.343 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:44.389 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:44.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:44.449 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:44.471+0000 7f4007844800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:44.449 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:44.472+0000 7f4007844800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:44.470 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:44Z"} 2024-03-04T16:56:44.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:44Z"} 2024-03-04T16:56:44.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:44.471 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:44.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:44.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T16:56:49.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:49.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:49.473 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:49.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:49.520 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:49.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:49.596+0000 7fb8ae88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:49.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:49.597+0000 7fb8ae88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:49.599 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:49Z"} 2024-03-04T16:56:49.600 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:49Z"} 2024-03-04T16:56:49.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:49.600 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:49.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:49.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:56:54.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:54.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:54.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:54.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:54.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:54.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:54.743+0000 7f2cbd11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:54.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:54.745+0000 7f2cbd11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:54.745 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:54Z"} 2024-03-04T16:56:54.745 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:54Z"} 2024-03-04T16:56:54.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:54.745 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:54.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:54.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:56:59.182 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T16:56:59.203+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T16:56:59.183 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T16:56:59.204+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T16:56:59.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:59.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:56:59.747 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:56:59.793 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:56:59.794 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:56:59.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:59.865+0000 7f535e12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:59.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:56:59.866+0000 7f535e12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:56:59.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:59Z"} 2024-03-04T16:56:59.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:59Z"} 2024-03-04T16:56:59.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:56:59.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:59.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:56:59.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T16:57:04.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:04.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:04.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:05.005+0000 7febecf26800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:04.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:05.006+0000 7febecf26800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:05.008 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:05Z"} 2024-03-04T16:57:05.008 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:05Z"} 2024-03-04T16:57:05.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:05.009 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:05.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:05.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:57:10.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:10.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:10.010 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:10.062 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:10.062 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:10.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:10.145+0000 7f6e62e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:10.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:10.145+0000 7f6e62e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:10.155 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:10Z"} 2024-03-04T16:57:10.155 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:10Z"} 2024-03-04T16:57:10.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:10.156 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:10.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:10.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:57:15.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:15.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:15.158 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:15.206 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:15.206 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:15.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:15.285+0000 7f24ede16800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:15.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:15.285+0000 7f24ede16800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:15.285 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:15Z"} 2024-03-04T16:57:15.285 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:15Z"} 2024-03-04T16:57:15.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:15.285 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:15.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:15.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T16:57:20.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:20.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:20.287 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:20.334 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:20.335 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:20.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:20.419+0000 7f8e51e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:20.397 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:20.420+0000 7f8e51e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:20.421 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:20Z"} 2024-03-04T16:57:20.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:20Z"} 2024-03-04T16:57:20.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:20.421 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:20.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:20.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T16:57:25.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:25.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:25.423 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:25.496 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:25.497 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:25.643 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:25.666+0000 7f5b5469d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:25.667+0000 7f5b5469d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:25.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:25Z"} 2024-03-04T16:57:25.676 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:25Z"} 2024-03-04T16:57:25.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:25.677 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:25.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:25.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:57:30.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:30.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:30.679 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:30.774 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:30.797+0000 7fec5ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:30.775 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:30.798+0000 7fec5ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:30.806 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:30Z"} 2024-03-04T16:57:30.806 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:30Z"} 2024-03-04T16:57:30.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:30.806 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:30.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:30.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T16:57:35.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:35.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:35.808 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:35.860 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:35.860 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:35.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:35.936+0000 7f907ae3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:35.937+0000 7f907ae3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:35.940 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:35Z"} 2024-03-04T16:57:35.941 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:35Z"} 2024-03-04T16:57:35.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:35.941 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:35.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:35.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T16:57:40.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:40.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:40.943 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:40.989 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:40.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:41.049 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:41.072+0000 7fd06c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:41.050 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:41.073+0000 7fd06c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:41.075 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:41Z"} 2024-03-04T16:57:41.075 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:41Z"} 2024-03-04T16:57:41.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:41.075 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:41.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:41.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T16:57:46.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:46.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:46.077 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:46.123 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:46.123 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:46.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:46.211+0000 7f270d214800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:46.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:46.212+0000 7f270d214800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:46.216 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:46Z"} 2024-03-04T16:57:46.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:46Z"} 2024-03-04T16:57:46.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:46.217 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:46.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:46.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:57:51.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:51.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:51.218 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:51.271 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:51.271 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:51.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:51.345+0000 7f22821a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:51.323 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:51.346+0000 7f22821a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:51.347 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:51Z"} 2024-03-04T16:57:51.347 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:51Z"} 2024-03-04T16:57:51.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:51.348 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:51.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:51.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T16:57:56.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:56.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:57:56.349 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:57:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:57:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:57:56.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:56.486+0000 7f3c7f667800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:56.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:57:56.487+0000 7f3c7f667800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:57:56.489 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:56Z"} 2024-03-04T16:57:56.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:56Z"} 2024-03-04T16:57:56.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:57:56.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:56.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:57:56.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:58:01.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:01.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:01.491 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:01.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:01.701+0000 7fca3bb30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:01.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:01.703+0000 7fca3bb30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:01.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:01Z"} 2024-03-04T16:58:01.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:01Z"} 2024-03-04T16:58:01.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:01.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:01.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:01.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:58:06.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:06.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:06.705 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:06.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:06.842+0000 7fb758630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:06.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:06.843+0000 7fb758630800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:06.843 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:06Z"} 2024-03-04T16:58:06.843 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:06Z"} 2024-03-04T16:58:06.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:06.844 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:06.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:06.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T16:58:11.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:11.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:11.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:11.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:11.980+0000 7f52b3044800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:11.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:11.981+0000 7f52b3044800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:11.989 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:11Z"} 2024-03-04T16:58:11.989 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:11Z"} 2024-03-04T16:58:11.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:11.990 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:11.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:11.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T16:58:16.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:16.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:16.992 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:17.039 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:17.039 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:17.111 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:17.134+0000 7f92d9739800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:17.112 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:17.135+0000 7f92d9739800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:17.133 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:17Z"} 2024-03-04T16:58:17.133 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:17Z"} 2024-03-04T16:58:17.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:17.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:17.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:17.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T16:58:22.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:22.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:22.136 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:22.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:22.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:22.269+0000 7f1de1413800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:22.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:22.270+0000 7f1de1413800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:22Z"} 2024-03-04T16:58:22.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:22Z"} 2024-03-04T16:58:22.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:22.275 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:22.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:22.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T16:58:27.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:27.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:27.279 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:27.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:27.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:27.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:27.489+0000 7fa21aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:27.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:27.493+0000 7fa21aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:27.522 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:27Z"} 2024-03-04T16:58:27.522 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:27Z"} 2024-03-04T16:58:27.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:27.523 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:27.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:27.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:58:32.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:32.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:32.524 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:32.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:32.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:32.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:32.647+0000 7f2bfa49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:32.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:32.648+0000 7f2bfa49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:32.652 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:32Z"} 2024-03-04T16:58:32.653 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:32Z"} 2024-03-04T16:58:32.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:32.653 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:32.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:32.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:58:37.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:37.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:37.655 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:37.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:37.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:37.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:37.787+0000 7f4488e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:37.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:37.787+0000 7f4488e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:37.797 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:37Z"} 2024-03-04T16:58:37.797 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:37Z"} 2024-03-04T16:58:37.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:37.797 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:37.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:37.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T16:58:42.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:42.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:42.799 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:42.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:42.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:42.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:42.937+0000 7fdd24711800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:42.914 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:42.937+0000 7fdd24711800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:42.938 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:42Z"} 2024-03-04T16:58:42.938 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:42Z"} 2024-03-04T16:58:42.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:42.939 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:42.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:42.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T16:58:47.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:47.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:47.941 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:47.988 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:47.988 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:48.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:48.059+0000 7f74d1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:48.037 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:48.060+0000 7f74d1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:48.067 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:48Z"} 2024-03-04T16:58:48.067 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:48Z"} 2024-03-04T16:58:48.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:48.068 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:48.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:48.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T16:58:53.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:53.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:53.069 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:53.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:53.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:53.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:53.187+0000 7f3a27a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:53.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:53.188+0000 7f3a27a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:53.186 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:53Z"} 2024-03-04T16:58:53.186 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:53Z"} 2024-03-04T16:58:53.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:53.186 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:53.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:53.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T16:58:58.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:58.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:58:58.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:58:58.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:58:58.236 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:58:58.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:58.308+0000 7fb2e014b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:58.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:58:58.309+0000 7fb2e014b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:58:58.307 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:58Z"} 2024-03-04T16:58:58.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:58Z"} 2024-03-04T16:58:58.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:58:58.307 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:58.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:58:58.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T16:59:03.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:03.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:03.310 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:03.449+0000 7f3730e38800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:03.427 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:03.450+0000 7f3730e38800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:03.450 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:03Z"} 2024-03-04T16:59:03.450 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:03Z"} 2024-03-04T16:59:03.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:03.451 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:03.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:03.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T16:59:08.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:08.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:08.452 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:08.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:08.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:08.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:08.593+0000 7fa2d2b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:08.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:08.594+0000 7fa2d2b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:08.593 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:08Z"} 2024-03-04T16:59:08.593 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:08Z"} 2024-03-04T16:59:08.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:08.593 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:08.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:08.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T16:59:13.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:13.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:13.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:13.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:13.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:13.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:13.725+0000 7fc387d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:13.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:13.726+0000 7fc387d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:13.729 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:13Z"} 2024-03-04T16:59:13.729 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:13Z"} 2024-03-04T16:59:13.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:13.730 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:13.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:13.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:59:18.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:18.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:18.731 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:18.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:18.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:18.901 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:18.924+0000 7f127c644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:18.902 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:18.926+0000 7f127c644800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:18.927 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:18Z"} 2024-03-04T16:59:18.927 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:18Z"} 2024-03-04T16:59:18.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:18.927 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:18.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:18.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T16:59:23.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:23.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:23.929 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:23.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:23.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:24.034 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:24.057+0000 7f9fe61a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:24.034 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:24.058+0000 7f9fe61a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:24.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:24Z"} 2024-03-04T16:59:24.056 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:24Z"} 2024-03-04T16:59:24.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:24.056 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:24.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:24.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.799310_424.1 2024-03-04T16:59:29.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:29.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:29.059 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:29.190+0000 7f033002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:29.191+0000 7f033002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:29.189 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:29Z"} 2024-03-04T16:59:29.189 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:29Z"} 2024-03-04T16:59:29.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:29.190 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:29.190 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:29.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T16:59:34.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:34.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:34.192 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:34.241 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:34.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:34.321+0000 7ffbcd89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:34.322+0000 7ffbcd89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:34.320 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:34Z"} 2024-03-04T16:59:34.320 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:34Z"} 2024-03-04T16:59:34.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:34.321 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:34.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:34.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.816633_428.1 2024-03-04T16:59:39.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:39.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:39.322 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:39.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:39.441+0000 7f408f726800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:39.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:39.442+0000 7f408f726800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:39.440 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:39Z"} 2024-03-04T16:59:39.441 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:39Z"} 2024-03-04T16:59:39.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:39.441 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:39.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:39.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T16:59:44.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:44.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:44.443 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:44.491 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:44.491 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:44.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:44.567+0000 7f58ddc30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:44.544 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:44.568+0000 7f58ddc30800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:44.568 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:44Z"} 2024-03-04T16:59:44.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:44Z"} 2024-03-04T16:59:44.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:44.568 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:44.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:44.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T16:59:49.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:49.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:49.571 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:49.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:49.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:49.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:49.705+0000 7f179e74b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:49.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:49.706+0000 7f179e74b800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:49.759 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:49Z"} 2024-03-04T16:59:49.760 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:49Z"} 2024-03-04T16:59:49.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:49.760 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:49.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:49.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.799310_424.1 2024-03-04T16:59:54.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:54.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:54.762 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:54.832 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:54.832 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T16:59:54.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:54.936+0000 7fb06769d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:54.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T16:59:54.937+0000 7fb06769d800 20 RGW-SYNC:meta: read sync status 2024-03-04T16:59:54.940 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:54Z"} 2024-03-04T16:59:54.940 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:54Z"} 2024-03-04T16:59:54.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T16:59:54.940 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:54.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T16:59:54.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.738966_421.1 2024-03-04T16:59:59.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:59.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T16:59:59.943 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T16:59:59.988 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T16:59:59.988 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:00.038 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:00.061+0000 7f627c33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:00.039 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:00.062+0000 7f627c33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:00.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:00Z"} 2024-03-04T17:00:00.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:00Z"} 2024-03-04T17:00:00.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:00.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:00.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:00.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T17:00:05.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:05.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:05.068 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:05.114 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:05.114 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:05.175 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:05.199+0000 7fb197331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:05.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:05.199+0000 7fb197331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:05.201 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:05Z"} 2024-03-04T17:00:05.201 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:05Z"} 2024-03-04T17:00:05.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:05.201 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:05.201 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:05.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:00:10.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:10.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:10.203 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:10.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:10.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:10.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:10.320+0000 7ff873c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:10.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:10.321+0000 7ff873c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:10.318 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:10Z"} 2024-03-04T17:00:10.319 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:10Z"} 2024-03-04T17:00:10.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.815644_427.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.815644Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:10.319 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:10.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.815644_427.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:10.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.815644_427.1 2024-03-04T17:00:15.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:15.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:15.321 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:15.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:15.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:15.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:15.457+0000 7f4fab920800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:15.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:15.458+0000 7f4fab920800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:15.462 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:15Z"} 2024-03-04T17:00:15.462 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:15Z"} 2024-03-04T17:00:15.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:15.462 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:15.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:15.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T17:00:20.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:20.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:20.464 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:20.517 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:20.517 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:20.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:20.681+0000 7fb161f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:20.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:20.684+0000 7fb161f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:20.729 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:20Z"} 2024-03-04T17:00:20.730 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:20Z"} 2024-03-04T17:00:20.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:20.730 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:20.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:20.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:00:25.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:25.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:25.732 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:25.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:25.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:25.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:25.872+0000 7f1c8052f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:25.849 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:25.873+0000 7f1c8052f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:25.882 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:25Z"} 2024-03-04T17:00:25.882 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:25Z"} 2024-03-04T17:00:25.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:25.883 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:25.883 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:25.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:00:30.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:30.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:30.884 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:30.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:30.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:30.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:31.017+0000 7fb9c8b3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:30.994 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:31.018+0000 7fb9c8b3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:31.025 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:31Z"} 2024-03-04T17:00:31.025 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:31Z"} 2024-03-04T17:00:31.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:31.026 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:31.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:31.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:00:36.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:36.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:36.027 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:36.074 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:36.074 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:36.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:36.143+0000 7fbe08da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:36.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:36.144+0000 7fbe08da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:36.141 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:36Z"} 2024-03-04T17:00:36.141 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:36Z"} 2024-03-04T17:00:36.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:36.142 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:36.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:36.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:00:41.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:41.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:41.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:41.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:41.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:41.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:41.268+0000 7f05f3644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:41.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:41.269+0000 7f05f3644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:41.267 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:41Z"} 2024-03-04T17:00:41.267 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:41Z"} 2024-03-04T17:00:41.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:41.268 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:41.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:41.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:00:46.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:46.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:46.270 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:46.321 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:46.321 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:46.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:46.416+0000 7f5f7062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:46.393 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:46.417+0000 7f5f7062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:46.421 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:46Z"} 2024-03-04T17:00:46.422 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:46Z"} 2024-03-04T17:00:46.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:46.422 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:46.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:46.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T17:00:51.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:51.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:51.424 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:51.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:51.479 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:51.631 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:51.655+0000 7f3d7892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:51.632 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:51.656+0000 7f3d7892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:51.672 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:51Z"} 2024-03-04T17:00:51.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:51Z"} 2024-03-04T17:00:51.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:51.673 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:51.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:51.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T17:00:56.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:56.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:00:56.675 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:00:56.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:00:56.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:00:56.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:56.812+0000 7f8d53511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:56.789 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:00:56.813+0000 7f8d53511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:00:56.819 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:56Z"} 2024-03-04T17:00:56.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:56Z"} 2024-03-04T17:00:56.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:00:56.820 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:56.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:00:56.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T17:01:01.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:01.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:01.822 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:01.871 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:01.871 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:01.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:01.954+0000 7f5373e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:01.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:01.955+0000 7f5373e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:01.955 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:01Z"} 2024-03-04T17:01:01.955 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:01Z"} 2024-03-04T17:01:01.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:01.955 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:01.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:01.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T17:01:06.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-03-04T17:01:06.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:06.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:06.961 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:01:07.088 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709571357.890340_643.1","last_update":"2024-03-04T16:55:57.890340Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:01:07.088 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:07.088 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:01:07.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:07.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:07.089 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:07.136 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:07.136 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:07.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:07.217+0000 7fccc8d2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:07.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:07.218+0000 7fccc8d2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:07.215 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:07Z"} 2024-03-04T17:01:07.215 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:07Z"} 2024-03-04T17:01:07.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:07.216 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:07.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:07.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:01:12.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:12.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:12.217 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:12.264 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:12.265 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:12.336+0000 7fe43180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:12.337+0000 7fe43180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:12.335 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:12Z"} 2024-03-04T17:01:12.335 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:12Z"} 2024-03-04T17:01:12.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:12.335 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:12.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:12.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T17:01:17.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:17.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:17.337 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:17.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:17.563+0000 7ff03a085800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:17.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:17.565+0000 7ff03a085800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:17.574 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:17Z"} 2024-03-04T17:01:17.575 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:17Z"} 2024-03-04T17:01:17.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:17.575 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:17.575 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:17.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T17:01:22.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:22.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:22.577 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:22.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:22.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:22.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:22.693+0000 7f3455733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:22.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:22.694+0000 7f3455733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:22Z"} 2024-03-04T17:01:22.691 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:22Z"} 2024-03-04T17:01:22.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:22.692 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:22.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:22.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:01:27.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:27.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:27.694 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:27.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:27.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:27.819+0000 7f677412b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:27.820+0000 7f677412b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:27.817 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:27Z"} 2024-03-04T17:01:27.818 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:27Z"} 2024-03-04T17:01:27.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:27.818 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:27.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:27.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:01:32.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:32.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:32.819 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:32.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:32.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:32.943 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:32.967+0000 7f97f9614800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:32.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:32.968+0000 7f97f9614800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:32.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:32Z"} 2024-03-04T17:01:32.971 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:32Z"} 2024-03-04T17:01:32.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:32.971 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:32.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:32.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:01:37.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:37.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:37.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:38.022 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:38.022 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:38.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:38.108+0000 7f2534b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:38.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:38.109+0000 7f2534b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:38.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:38Z"} 2024-03-04T17:01:38.107 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:38Z"} 2024-03-04T17:01:38.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:38.107 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:38.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:38.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T17:01:43.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:43.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:43.109 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:43.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:43.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:43.254+0000 7fa25e12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:43.255+0000 7fa25e12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:43.258 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:43Z"} 2024-03-04T17:01:43.258 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:43Z"} 2024-03-04T17:01:43.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:43.259 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:43.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:43.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:01:48.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:48.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:48.260 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:48.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:48.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:48.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:48.484+0000 7fc9c232b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:48.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:48.485+0000 7fc9c232b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:48.485 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:48Z"} 2024-03-04T17:01:48.485 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:48Z"} 2024-03-04T17:01:48.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:48.486 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:48.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:48.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:01:49.202 INFO:tasks.rgw.c1.client.1.smithi027.stdout:2024-03-04T17:01:49.226+0000 7fc935f0d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:01:53.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:53.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:53.487 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:53.537 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:53.537 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:53.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:53.622+0000 7faa93c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:53.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:53.623+0000 7faa93c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:53.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:53Z"} 2024-03-04T17:01:53.630 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:53Z"} 2024-03-04T17:01:53.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:53.630 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:53.631 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:53.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:01:58.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:58.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:01:58.632 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:01:58.678 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:01:58.679 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:01:58.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:58.756+0000 7fed1ca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:58.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:01:58.757+0000 7fed1ca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:01:58.755 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:58Z"} 2024-03-04T17:01:58.755 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:58Z"} 2024-03-04T17:01:58.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:01:58.755 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:58.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:01:58.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.948368_440.1 2024-03-04T17:02:03.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:03.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:03.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:03.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:03.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:03.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:03.888+0000 7f9568a3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:03.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:03.889+0000 7f9568a3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:03.886 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:03Z"} 2024-03-04T17:02:03.887 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:03Z"} 2024-03-04T17:02:03.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:03.888 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:03.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:03.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T17:02:08.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:08.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:08.889 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:08.942 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:08.942 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:08.994 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:09.019+0000 7f4ca2b39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:08.995 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:09.020+0000 7f4ca2b39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:09.018 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:09Z"} 2024-03-04T17:02:09.018 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:09Z"} 2024-03-04T17:02:09.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:09.018 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:09.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:09.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:02:14.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:14.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:14.019 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:14.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:14.172+0000 7f57d4ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:14.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:14.173+0000 7f57d4ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:14.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:14Z"} 2024-03-04T17:02:14.179 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:14Z"} 2024-03-04T17:02:14.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:14.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:14.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:14.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:02:19.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:19.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:19.181 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:19.238 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:02:19.262+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:02:19.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:19.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:19.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:19.321+0000 7fd8de33c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:19.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:19.322+0000 7fd8de33c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:19.324 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:19Z"} 2024-03-04T17:02:19.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:19Z"} 2024-03-04T17:02:19.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:19.325 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:19.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:19.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:02:24.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:24.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:24.330 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:24.409 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:24.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:24.527+0000 7f5f60111800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:24.503 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:24.528+0000 7f5f60111800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:24.535 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:24Z"} 2024-03-04T17:02:24.536 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:24Z"} 2024-03-04T17:02:24.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:24.536 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:24.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:24.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.950988_441.1 2024-03-04T17:02:29.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:29.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:29.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:29.585 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:29.585 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:29.675+0000 7fbc9b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:29.676+0000 7fbc9b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:29Z"} 2024-03-04T17:02:29.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:29Z"} 2024-03-04T17:02:29.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:29.674 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:29.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:29.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:02:34.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:34.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:34.676 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:34.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:34.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:34.790 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:34.814+0000 7f3e9e91f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:34.791 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:34.815+0000 7f3e9e91f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:34.817 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:34Z"} 2024-03-04T17:02:34.817 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:34Z"} 2024-03-04T17:02:34.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:34.818 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:34.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:34.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:02:39.181 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:02:39.206+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:02:39.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:39.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:39.820 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:39.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:39.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:39.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:39.952+0000 7f020c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:39.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:39.953+0000 7f020c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:39.953 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:39Z"} 2024-03-04T17:02:39.953 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:39Z"} 2024-03-04T17:02:39.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:39.954 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:39.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:39.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:02:44.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:44.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:44.956 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:45.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:45.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:45.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:45.097+0000 7f8d33f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:45.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:45.098+0000 7f8d33f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:45.104 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:45Z"} 2024-03-04T17:02:45.105 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:45Z"} 2024-03-04T17:02:45.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:45.105 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:45.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:45.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:02:50.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:50.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:50.107 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:50.159 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:50.159 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:50.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:50.354+0000 7f0e42e15800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:50.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:50.357+0000 7f0e42e15800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:50.364 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:50Z"} 2024-03-04T17:02:50.365 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:50Z"} 2024-03-04T17:02:50.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:50.365 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:50.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:50.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:02:55.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:55.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:02:55.367 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:02:55.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:02:55.414 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:02:55.486 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:55.511+0000 7f5600424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:55.487 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:02:55.512+0000 7f5600424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:02:55.512 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:55Z"} 2024-03-04T17:02:55.512 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:55Z"} 2024-03-04T17:02:55.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:02:55.512 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:55.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:02:55.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:03:00.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:00.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:00.515 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:00.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:00.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:00.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:00.634+0000 7f6f48667800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:00.610 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:00.635+0000 7f6f48667800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:00.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:00Z"} 2024-03-04T17:03:00.633 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:00Z"} 2024-03-04T17:03:00.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:00.633 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:00.633 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:00.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:03:05.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:05.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:05.635 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:05.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:05.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:05.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:05.760+0000 7fb26de12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:05.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:05.761+0000 7fb26de12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:05.762 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:05Z"} 2024-03-04T17:03:05.762 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:05Z"} 2024-03-04T17:03:05.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:05.763 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:05.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:05.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T17:03:10.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:10.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:10.764 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:10.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:10.813 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:10.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:10.904+0000 7fe4a4e32800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:10.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:10.904+0000 7fe4a4e32800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:10.901 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:10Z"} 2024-03-04T17:03:10.901 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:10Z"} 2024-03-04T17:03:10.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:10.902 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:10.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:10.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T17:03:15.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:15.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:15.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:15.961 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:15.962 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:16.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:16.100+0000 7f2894e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:16.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:16.101+0000 7f2894e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:16.105 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:16Z"} 2024-03-04T17:03:16.105 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:16Z"} 2024-03-04T17:03:16.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:16.106 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:16.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:16.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T17:03:21.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:21.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:21.109 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:21.186 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:21.186 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:21.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:21.337+0000 7fb7b4613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:21.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:21.338+0000 7fb7b4613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:21.336 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:21Z"} 2024-03-04T17:03:21.336 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:21Z"} 2024-03-04T17:03:21.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:21.337 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:21.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:21.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:03:26.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:26.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:26.338 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:26.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:26.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:26.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:26.469+0000 7f0fd313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:26.470+0000 7f0fd313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:26.470 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:26Z"} 2024-03-04T17:03:26.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:26Z"} 2024-03-04T17:03:26.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:26.471 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:26.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:26.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:03:31.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:31.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:31.473 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:31.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:31.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:31.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:31.593+0000 7fead4414800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:31.594+0000 7fead4414800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:31.592 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:31Z"} 2024-03-04T17:03:31.592 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:31Z"} 2024-03-04T17:03:31.592 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:31.593 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:31.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:31.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:03:36.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:36.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:36.594 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:36.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:36.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:36.737+0000 7f62c482f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:36.714 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:36.738+0000 7f62c482f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:36.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:36Z"} 2024-03-04T17:03:36.738 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:36Z"} 2024-03-04T17:03:36.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:36.738 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:36.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:36.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T17:03:41.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:41.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:41.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:41.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:41.790 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:41.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:41.874+0000 7f4861430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:41.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:41.875+0000 7f4861430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:41.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:41Z"} 2024-03-04T17:03:41.877 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:41Z"} 2024-03-04T17:03:41.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:41.877 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:41.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:41.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T17:03:46.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:46.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:46.879 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:46.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:46.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:46.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:47.004+0000 7fd76802e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:46.982 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:47.005+0000 7fd76802e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:47.003 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:47Z"} 2024-03-04T17:03:47.003 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:47Z"} 2024-03-04T17:03:47.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:47.004 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:47.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:47.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:03:52.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:52.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:52.005 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:52.057 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:52.057 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:52.131 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:52.154+0000 7f94959a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:52.132 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:52.155+0000 7f94959a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:52.154 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:52Z"} 2024-03-04T17:03:52.154 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:52Z"} 2024-03-04T17:03:52.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:52.154 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:52.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:52.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T17:03:57.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:57.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:03:57.157 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:03:57.210 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:03:57.210 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:03:57.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:57.287+0000 7f3566a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:57.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:03:57.288+0000 7f3566a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:03:57.332 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:57Z"} 2024-03-04T17:03:57.332 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:57Z"} 2024-03-04T17:03:57.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:03:57.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:57.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:03:57.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:04:02.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:02.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:02.334 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:02.409 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:02.409 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:02.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:02.489+0000 7ff4a972a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:02.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:02.491+0000 7ff4a972a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:02.490 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:02Z"} 2024-03-04T17:04:02.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:02Z"} 2024-03-04T17:04:02.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:02.491 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:02.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:02.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T17:04:07.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:07.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:07.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:07.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:07.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:07.589 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:07.613+0000 7f1a99a17800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:07.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:07.613+0000 7f1a99a17800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:07.612 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:07Z"} 2024-03-04T17:04:07.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:07Z"} 2024-03-04T17:04:07.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:07.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:07.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:07.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T17:04:12.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:12.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:12.614 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:12.660 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:12.660 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:12.746+0000 7fbe1c539800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:12.747+0000 7fbe1c539800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:12.750 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:12Z"} 2024-03-04T17:04:12.750 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:12Z"} 2024-03-04T17:04:12.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:12.751 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:12.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:12.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.738966_421.1 2024-03-04T17:04:17.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:17.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:17.753 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:17.802 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:17.802 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:17.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:17.879+0000 7eff61d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:17.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:17.881+0000 7eff61d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:17.880 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:17Z"} 2024-03-04T17:04:17.881 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:17Z"} 2024-03-04T17:04:17.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:17.881 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:17.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:17.881 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:04:22.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:22.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:22.883 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:22.935 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:22.936 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:23.004 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:23.026+0000 7f1be2267800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:23.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:23.027+0000 7f1be2267800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:23.034 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:23Z"} 2024-03-04T17:04:23.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:23Z"} 2024-03-04T17:04:23.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:23.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:23.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:23.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:04:28.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:28.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:28.036 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:28.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:28.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:28.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:28.160+0000 7f3a97ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:28.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:28.161+0000 7f3a97ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:28.164 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:28Z"} 2024-03-04T17:04:28.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:28Z"} 2024-03-04T17:04:28.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:28.165 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:28.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:28.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T17:04:33.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:33.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:33.167 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:33.220 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:33.221 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:33.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:33.301+0000 7f8f88624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:33.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:33.301+0000 7f8f88624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:33.310 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:33Z"} 2024-03-04T17:04:33.310 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:33Z"} 2024-03-04T17:04:33.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:33.310 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:33.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:33.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:04:38.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:38.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:38.312 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:38.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:38.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:38.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:38.526+0000 7fc1f3e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:38.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:38.527+0000 7fc1f3e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:38.531 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:38Z"} 2024-03-04T17:04:38.531 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:38Z"} 2024-03-04T17:04:38.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:38.532 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:38.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:38.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T17:04:43.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:43.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:43.534 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:43.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:43.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:43.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:43.665+0000 7f5ad1d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:43.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:43.666+0000 7f5ad1d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:43.675 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:43Z"} 2024-03-04T17:04:43.676 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:43Z"} 2024-03-04T17:04:43.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:43.676 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:43.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:43.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T17:04:48.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:48.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:48.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:48.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:48.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:48.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:48.808+0000 7f595480f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:48.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:48.809+0000 7f595480f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:48.815 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:48Z"} 2024-03-04T17:04:48.815 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:48Z"} 2024-03-04T17:04:48.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:48.815 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:48.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:48.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.858235_434.1 2024-03-04T17:04:53.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:53.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:53.817 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:53.866 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:53.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:53.916 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:53.937+0000 7f4818e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:53.917 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:53.938+0000 7f4818e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:53.941 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:53Z"} 2024-03-04T17:04:53.941 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:53Z"} 2024-03-04T17:04:53.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:53.941 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:53.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:53.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:04:58.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:58.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:04:58.944 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:04:58.993 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:04:58.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:04:59.052 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:59.073+0000 7f4fd980f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:59.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:04:59.074+0000 7f4fd980f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:04:59.081 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:59Z"} 2024-03-04T17:04:59.082 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:59Z"} 2024-03-04T17:04:59.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:04:59.082 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:59.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:04:59.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T17:05:04.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:04.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:04.084 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:04.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:04.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:04.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:04.209+0000 7fe63ea16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:04.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:04.210+0000 7fe63ea16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:04.216 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:04Z"} 2024-03-04T17:05:04.216 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:04Z"} 2024-03-04T17:05:04.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:04.217 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:04.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:04.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.951904_442.1 2024-03-04T17:05:09.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:09.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:09.219 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:09.272 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:09.272 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:09.325 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:09.345+0000 7f7e74c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:09.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:09.346+0000 7f7e74c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:09.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:09Z"} 2024-03-04T17:05:09.351 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:09Z"} 2024-03-04T17:05:09.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:09.351 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:09.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:09.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:05:14.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:14.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:14.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:14.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:14.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:14.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:14.571+0000 7fb748a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:14.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:14.576+0000 7fb748a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:14.601 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:14Z"} 2024-03-04T17:05:14.601 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:14Z"} 2024-03-04T17:05:14.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:14.602 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:14.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:14.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T17:05:19.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:19.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:19.603 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:19.650 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:19.650 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:19.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:19.732+0000 7f213e4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:19.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:19.733+0000 7f213e4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:19.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:19Z"} 2024-03-04T17:05:19.738 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:19Z"} 2024-03-04T17:05:19.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:19.738 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:19.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:19.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:05:24.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:24.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:24.740 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:24.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:24.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:24.837 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:24.856+0000 7f736c333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:24.838 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:24.857+0000 7f736c333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:24.859 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:24Z"} 2024-03-04T17:05:24.859 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:24Z"} 2024-03-04T17:05:24.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:24.860 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:24.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:24.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:05:29.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:29.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:29.861 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:29.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:29.987+0000 7f994622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:29.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:29.988+0000 7f994622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:29.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:29Z"} 2024-03-04T17:05:29.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:29Z"} 2024-03-04T17:05:29.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:29.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:29.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:29.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T17:05:34.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:35.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:35.000 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:35.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:35.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:35.117 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:35.136+0000 7f0e91333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:35.137+0000 7f0e91333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:35.147 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:35Z"} 2024-03-04T17:05:35.147 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:35Z"} 2024-03-04T17:05:35.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.819018_429.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.819018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:35.148 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:35.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.819018_429.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:35.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.819018_429.1 2024-03-04T17:05:40.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:40.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:40.149 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:40.230 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:40.230 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:40.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:40.324+0000 7f31b3d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:40.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:40.325+0000 7f31b3d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:40.329 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:40Z"} 2024-03-04T17:05:40.329 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:40Z"} 2024-03-04T17:05:40.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:40.329 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:40.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:40.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T17:05:45.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:45.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:45.331 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:45.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:45.451+0000 7fcb43c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:45.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:45.452+0000 7fcb43c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:45.455 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:45Z"} 2024-03-04T17:05:45.455 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:45Z"} 2024-03-04T17:05:45.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:45.456 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:45.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:45.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:05:50.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:50.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:50.458 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:50.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:50.507 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:50.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:50.585+0000 7fa172f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:50.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:50.586+0000 7fa172f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:50.594 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:50Z"} 2024-03-04T17:05:50.594 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:50Z"} 2024-03-04T17:05:50.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:50.595 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:50.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:50.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T17:05:55.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:55.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:05:55.596 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:05:55.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:05:55.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:05:55.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:55.731+0000 7fa2f5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:55.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:05:55.731+0000 7fa2f5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:05:55.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:55Z"} 2024-03-04T17:05:55.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:55Z"} 2024-03-04T17:05:55.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:05:55.738 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:55.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:05:55.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.927093_438.1 2024-03-04T17:06:00.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:00.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:00.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:00.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:00.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:00.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:01.001+0000 7f7faef1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:00.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:01.017+0000 7f7faef1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:01.094 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:01Z"} 2024-03-04T17:06:01.095 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:01Z"} 2024-03-04T17:06:01.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:01.096 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:01.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:01.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T17:06:06.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:06.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:06.097 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:06.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:06.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:06.458 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:06.476+0000 7f4e73816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:06.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:06.477+0000 7f4e73816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:06.483 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:06Z"} 2024-03-04T17:06:06.483 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:06Z"} 2024-03-04T17:06:06.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:06.483 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:06.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:06.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T17:06:11.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:11.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:11.488 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:11.535 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:11.536 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:11.593 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:11.610+0000 7f4c75430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:11.594 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:11.611+0000 7f4c75430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:11.615 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:11Z"} 2024-03-04T17:06:11.616 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:11Z"} 2024-03-04T17:06:11.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:11.616 DEBUG:rgw_multi.tests:log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:11.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:11.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T17:06:16.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2024-03-04T17:06:16.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.621 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:16.672 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:16.672 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"ec7b3b08-66d3-4c1c-8b9e-58b7f9e916e6","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]} 2024-03-04T17:06:16.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.720 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:16.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T17:06:16.823+0000 7f6453a12800 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-03-04T17:06:16.817 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"c8f1fffc-3dca-43d6-8002-60b11288ea44","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"remove.rgw.log:notif"} 2024-03-04T17:06:16.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.817 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:16.952 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":5,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"ec7b3b08-66d3-4c1c-8b9e-58b7f9e916e6","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"c8f1fffc-3dca-43d6-8002-60b11288ea44","zones":[{"id":"c8f1fffc-3dca-43d6-8002-60b11288ea44","name":"remove","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","endpoints":["http://smithi027.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"c8f1fffc-3dca-43d6-8002-60b11288ea44","val":2408928080},{"key":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","val":210978822},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T17:06:16.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:16.952 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:16.996 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:16.996 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:17.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T17:06:17.051+0000 7f6d40412800 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-03-04T17:06:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T17:06:17.052+0000 7f6d40412800 0 Removed zone from zonegroup remove_zg 2024-03-04T17:06:17.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.048 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-04T17:06:17.157+0000 7f331d22d800 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-03-04T17:06:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T17:06:17.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.156 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:17.219 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:17.219 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:17.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T17:06:17.314 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-03-04T17:06:17.387 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:17.387 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:17.559 INFO:teuthology.orchestra.run.smithi027.stdout:{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":6,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","endpoints":["http://smithi027.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","val":210978822},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T17:06:17.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-03-04T17:06:17.560 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-49 2024-03-04T17:06:17.560 DEBUG:boto:path=/sbnvju-49/ 2024-03-04T17:06:17.560 DEBUG:boto:auth_path=/sbnvju-49/ 2024-03-04T17:06:17.561 DEBUG:boto:Method: PUT 2024-03-04T17:06:17.561 DEBUG:boto:Path: /sbnvju-49/ 2024-03-04T17:06:17.561 DEBUG:boto:Data: 2024-03-04T17:06:17.561 DEBUG:boto:Headers: {} 2024-03-04T17:06:17.561 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:17.561 DEBUG:boto:Port: 8000 2024-03-04T17:06:17.561 DEBUG:boto:Params: {} 2024-03-04T17:06:17.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:06:17.562 DEBUG:boto:Token: None 2024-03-04T17:06:17.562 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:17 GMT /sbnvju-49/ 2024-03-04T17:06:17.562 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:opY+9QswMUPNLpCzuj9fuJmnsAQ= 2024-03-04T17:06:17.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:opY+9QswMUPNLpCzuj9fuJmnsAQ=', 'Content-Length': '0'} 2024-03-04T17:06:17.828 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009fbabf34bbd5d3b4-0065e5ff89-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:17.828 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-50 2024-03-04T17:06:17.828 DEBUG:boto:path=/sbnvju-50/ 2024-03-04T17:06:17.828 DEBUG:boto:auth_path=/sbnvju-50/ 2024-03-04T17:06:17.828 DEBUG:boto:Method: PUT 2024-03-04T17:06:17.828 DEBUG:boto:Path: /sbnvju-50/ 2024-03-04T17:06:17.828 DEBUG:boto:Data: 2024-03-04T17:06:17.829 DEBUG:boto:Headers: {} 2024-03-04T17:06:17.829 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:06:17.829 DEBUG:boto:Port: 8000 2024-03-04T17:06:17.829 DEBUG:boto:Params: {} 2024-03-04T17:06:17.829 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:06:17.829 DEBUG:boto:Token: None 2024-03-04T17:06:17.829 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:17 GMT /sbnvju-50/ 2024-03-04T17:06:17.829 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aVKbNYNdCd5/udA1TUCbyRsr17U= 2024-03-04T17:06:17.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aVKbNYNdCd5/udA1TUCbyRsr17U=', 'Content-Length': '0'} 2024-03-04T17:06:18.084 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d569ad8f44600560-0065e5ff89-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.084 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-51 2024-03-04T17:06:18.084 DEBUG:boto:path=/sbnvju-51/ 2024-03-04T17:06:18.084 DEBUG:boto:auth_path=/sbnvju-51/ 2024-03-04T17:06:18.085 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.085 DEBUG:boto:Path: /sbnvju-51/ 2024-03-04T17:06:18.085 DEBUG:boto:Data: 2024-03-04T17:06:18.085 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.085 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:06:18.085 DEBUG:boto:Port: 8001 2024-03-04T17:06:18.085 DEBUG:boto:Params: {} 2024-03-04T17:06:18.085 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:06:18.085 DEBUG:boto:Token: None 2024-03-04T17:06:18.086 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-51/ 2024-03-04T17:06:18.086 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NhJFRqtRFLYGwRZQ+FhRoyMQdrw= 2024-03-04T17:06:18.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NhJFRqtRFLYGwRZQ+FhRoyMQdrw=', 'Content-Length': '0'} 2024-03-04T17:06:18.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026b3e67b3ef23b8c-0065e5ff8a-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.120 DEBUG:boto:path=/sbnvju-49/ 2024-03-04T17:06:18.120 DEBUG:boto:auth_path=/sbnvju-49/ 2024-03-04T17:06:18.120 DEBUG:boto:path=/sbnvju-49/?website 2024-03-04T17:06:18.120 DEBUG:boto:auth_path=/sbnvju-49/?website 2024-03-04T17:06:18.120 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.120 DEBUG:boto:Path: /sbnvju-49/?website 2024-03-04T17:06:18.120 DEBUG:boto:Data: index.htmlerror.html 2024-03-04T17:06:18.120 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.120 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:18.120 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.120 DEBUG:boto:Params: {} 2024-03-04T17:06:18.121 DEBUG:boto:Token: None 2024-03-04T17:06:18.121 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-49/?website 2024-03-04T17:06:18.121 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nU26pnnIeaCeFJW5rW1Cfw/nCu4= 2024-03-04T17:06:18.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nU26pnnIeaCeFJW5rW1Cfw/nCu4=', 'Content-Length': '241'} 2024-03-04T17:06:18.122 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000000aee9deac3c8f225-0065e5ff8a-5355-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-03-04T17:06:18.123 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-52 2024-03-04T17:06:18.123 DEBUG:boto:path=/sbnvju-52/ 2024-03-04T17:06:18.123 DEBUG:boto:auth_path=/sbnvju-52/ 2024-03-04T17:06:18.123 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.123 DEBUG:boto:Path: /sbnvju-52/ 2024-03-04T17:06:18.123 DEBUG:boto:Data: 2024-03-04T17:06:18.123 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.124 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:18.124 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.124 DEBUG:boto:Params: {} 2024-03-04T17:06:18.124 DEBUG:boto:Token: None 2024-03-04T17:06:18.124 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-52/ 2024-03-04T17:06:18.124 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:7b4S/+fZLl6jo4KESwIHBLB2vbM= 2024-03-04T17:06:18.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7b4S/+fZLl6jo4KESwIHBLB2vbM=', 'Content-Length': '0'} 2024-03-04T17:06:18.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bf229b95f5b509e-0065e5ff8a-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.143 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-53 2024-03-04T17:06:18.143 DEBUG:boto:path=/sbnvju-53/ 2024-03-04T17:06:18.144 DEBUG:boto:auth_path=/sbnvju-53/ 2024-03-04T17:06:18.144 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.144 DEBUG:boto:Path: /sbnvju-53/ 2024-03-04T17:06:18.144 DEBUG:boto:Data: 2024-03-04T17:06:18.144 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.144 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:06:18.144 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.144 DEBUG:boto:Params: {} 2024-03-04T17:06:18.144 DEBUG:boto:Token: None 2024-03-04T17:06:18.144 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-53/ 2024-03-04T17:06:18.145 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:L+FQySg1sYzg4S76T1ltboN1Uzg= 2024-03-04T17:06:18.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:L+FQySg1sYzg4S76T1ltboN1Uzg=', 'Content-Length': '0'} 2024-03-04T17:06:18.185 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000017cdd1deb3e42187-0065e5ff8a-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.185 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-54 2024-03-04T17:06:18.185 DEBUG:boto:path=/sbnvju-54/ 2024-03-04T17:06:18.185 DEBUG:boto:auth_path=/sbnvju-54/ 2024-03-04T17:06:18.185 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.185 DEBUG:boto:Path: /sbnvju-54/ 2024-03-04T17:06:18.185 DEBUG:boto:Data: 2024-03-04T17:06:18.185 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.186 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:06:18.186 DEBUG:boto:Port: 8001 2024-03-04T17:06:18.186 DEBUG:boto:Params: {} 2024-03-04T17:06:18.186 DEBUG:boto:Token: None 2024-03-04T17:06:18.186 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-54/ 2024-03-04T17:06:18.186 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pumSfoIrk6nFAoNWgORUf30ysC8= 2024-03-04T17:06:18.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pumSfoIrk6nFAoNWgORUf30ysC8=', 'Content-Length': '0'} 2024-03-04T17:06:18.221 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8c110aa9359d1aa-0065e5ff8a-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.221 DEBUG:boto:path=/sbnvju-52/ 2024-03-04T17:06:18.221 DEBUG:boto:auth_path=/sbnvju-52/ 2024-03-04T17:06:18.221 DEBUG:boto:path=/sbnvju-52/?policy 2024-03-04T17:06:18.221 DEBUG:boto:auth_path=/sbnvju-52/?policy 2024-03-04T17:06:18.221 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.221 DEBUG:boto:Path: /sbnvju-52/?policy 2024-03-04T17:06:18.221 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T17:06:18.221 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.221 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:18.221 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.222 DEBUG:boto:Params: {} 2024-03-04T17:06:18.222 DEBUG:boto:Token: None 2024-03-04T17:06:18.222 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-52/?policy 2024-03-04T17:06:18.222 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sBz2hY2YBKNPI18bUfkli/QQgSQ= 2024-03-04T17:06:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sBz2hY2YBKNPI18bUfkli/QQgSQ=', 'Content-Length': '97'} 2024-03-04T17:06:18.228 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa270236a56b42f9-0065e5ff8a-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.228 DEBUG:boto:path=/sbnvju-52/ 2024-03-04T17:06:18.228 DEBUG:boto:auth_path=/sbnvju-52/ 2024-03-04T17:06:18.228 DEBUG:boto:path=/sbnvju-52/?policy 2024-03-04T17:06:18.228 DEBUG:boto:auth_path=/sbnvju-52/?policy 2024-03-04T17:06:18.228 DEBUG:boto:Method: GET 2024-03-04T17:06:18.228 DEBUG:boto:Path: /sbnvju-52/?policy 2024-03-04T17:06:18.228 DEBUG:boto:Data: 2024-03-04T17:06:18.229 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.229 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:18.229 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.229 DEBUG:boto:Params: {} 2024-03-04T17:06:18.229 DEBUG:boto:Token: None 2024-03-04T17:06:18.229 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-52/?policy 2024-03-04T17:06:18.229 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:SYg4SPUd4sFlSdxDPb04Rojgqzc= 2024-03-04T17:06:18.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SYg4SPUd4sFlSdxDPb04Rojgqzc=', 'Content-Length': '0'} 2024-03-04T17:06:18.230 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de4942bc3e57347c-0065e5ff8a-5355-test-zone1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 17:06:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:18.231 DEBUG:boto:path=/sbnvju-53/ 2024-03-04T17:06:18.231 DEBUG:boto:auth_path=/sbnvju-53/ 2024-03-04T17:06:18.231 DEBUG:boto:path=/sbnvju-53/?policy 2024-03-04T17:06:18.231 DEBUG:boto:auth_path=/sbnvju-53/?policy 2024-03-04T17:06:18.231 DEBUG:boto:Method: PUT 2024-03-04T17:06:18.231 DEBUG:boto:Path: /sbnvju-53/?policy 2024-03-04T17:06:18.231 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T17:06:18.231 DEBUG:boto:Headers: {} 2024-03-04T17:06:18.231 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:06:18.231 DEBUG:boto:Port: 8000 2024-03-04T17:06:18.231 DEBUG:boto:Params: {} 2024-03-04T17:06:18.232 DEBUG:boto:Token: None 2024-03-04T17:06:18.232 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:18 GMT /sbnvju-53/?policy 2024-03-04T17:06:18.232 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:bqDX5qdx/v/GvSgMdFYjv3P60Xo= 2024-03-04T17:06:18.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:bqDX5qdx/v/GvSgMdFYjv3P60Xo=', 'Content-Length': '97'} 2024-03-04T17:06:19.270 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac8d3121a0f7165d-0065e5ff8a-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:19.270 DEBUG:boto:path=/sbnvju-53/ 2024-03-04T17:06:19.270 DEBUG:boto:auth_path=/sbnvju-53/ 2024-03-04T17:06:19.270 DEBUG:boto:path=/sbnvju-53/?policy 2024-03-04T17:06:19.270 DEBUG:boto:auth_path=/sbnvju-53/?policy 2024-03-04T17:06:19.271 DEBUG:boto:Method: GET 2024-03-04T17:06:19.271 DEBUG:boto:Path: /sbnvju-53/?policy 2024-03-04T17:06:19.271 DEBUG:boto:Data: 2024-03-04T17:06:19.271 DEBUG:boto:Headers: {} 2024-03-04T17:06:19.271 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:06:19.271 DEBUG:boto:Port: 8000 2024-03-04T17:06:19.271 DEBUG:boto:Params: {} 2024-03-04T17:06:19.271 DEBUG:boto:Token: None 2024-03-04T17:06:19.271 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:06:19 GMT /sbnvju-53/?policy 2024-03-04T17:06:19.272 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vzgsOdkEINcs2t217nRVf2FHPfY= 2024-03-04T17:06:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vzgsOdkEINcs2t217nRVf2FHPfY=', 'Content-Length': '0'} 2024-03-04T17:06:19.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb68f042df4b9062-0065e5ff8b-7072-test-zone2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 17:06:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:19.273 DEBUG:boto:path=/sbnvju-54/ 2024-03-04T17:06:19.273 DEBUG:boto:auth_path=/sbnvju-54/ 2024-03-04T17:06:19.273 DEBUG:boto:path=/sbnvju-54/?policy 2024-03-04T17:06:19.274 DEBUG:boto:auth_path=/sbnvju-54/?policy 2024-03-04T17:06:19.274 DEBUG:boto:Method: PUT 2024-03-04T17:06:19.274 DEBUG:boto:Path: /sbnvju-54/?policy 2024-03-04T17:06:19.274 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-03-04T17:06:19.274 DEBUG:boto:Headers: {} 2024-03-04T17:06:19.274 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:06:19.274 DEBUG:boto:Port: 8001 2024-03-04T17:06:19.274 DEBUG:boto:Params: {} 2024-03-04T17:06:19.274 DEBUG:boto:Token: None 2024-03-04T17:06:19.275 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:19 GMT /sbnvju-54/?policy 2024-03-04T17:06:19.275 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:1dzW4v+yH8f5v+Kys1zaXc3b134= 2024-03-04T17:06:19.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1dzW4v+yH8f5v+Kys1zaXc3b134=', 'Content-Length': '97'} 2024-03-04T17:06:20.286 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000018433048652168fe-0065e5ff8b-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:20.287 DEBUG:boto:path=/sbnvju-54/ 2024-03-04T17:06:20.287 DEBUG:boto:auth_path=/sbnvju-54/ 2024-03-04T17:06:20.287 DEBUG:boto:path=/sbnvju-54/?policy 2024-03-04T17:06:20.287 DEBUG:boto:auth_path=/sbnvju-54/?policy 2024-03-04T17:06:20.287 DEBUG:boto:Method: GET 2024-03-04T17:06:20.287 DEBUG:boto:Path: /sbnvju-54/?policy 2024-03-04T17:06:20.287 DEBUG:boto:Data: 2024-03-04T17:06:20.288 DEBUG:boto:Headers: {} 2024-03-04T17:06:20.288 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:06:20.288 DEBUG:boto:Port: 8001 2024-03-04T17:06:20.288 DEBUG:boto:Params: {} 2024-03-04T17:06:20.288 DEBUG:boto:Token: None 2024-03-04T17:06:20.288 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:06:20 GMT /sbnvju-54/?policy 2024-03-04T17:06:20.288 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MBkO1GugFWMcZL3cFAU0/gGYpIg= 2024-03-04T17:06:20.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MBkO1GugFWMcZL3cFAU0/gGYpIg=', 'Content-Length': '0'} 2024-03-04T17:06:20.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd33a1799c91f11f-0065e5ff8c-5358-test-zone3'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:20.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-03-04T17:06:20.290 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-55 2024-03-04T17:06:20.291 DEBUG:boto:path=/sbnvju-55/ 2024-03-04T17:06:20.291 DEBUG:boto:auth_path=/sbnvju-55/ 2024-03-04T17:06:20.291 DEBUG:boto:Method: PUT 2024-03-04T17:06:20.291 DEBUG:boto:Path: /sbnvju-55/ 2024-03-04T17:06:20.291 DEBUG:boto:Data: 2024-03-04T17:06:20.291 DEBUG:boto:Headers: {} 2024-03-04T17:06:20.291 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:06:20.291 DEBUG:boto:Port: 8000 2024-03-04T17:06:20.291 DEBUG:boto:Params: {} 2024-03-04T17:06:20.291 DEBUG:boto:Token: None 2024-03-04T17:06:20.292 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:20 GMT /sbnvju-55/ 2024-03-04T17:06:20.292 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mndBUIiPUKa5Fwbir/0gIXzhTsk= 2024-03-04T17:06:20.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mndBUIiPUKa5Fwbir/0gIXzhTsk=', 'Content-Length': '0'} 2024-03-04T17:06:20.311 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000636efaf50881980e-0065e5ff8c-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:20.311 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-56 2024-03-04T17:06:20.311 DEBUG:boto:path=/sbnvju-56/ 2024-03-04T17:06:20.311 DEBUG:boto:auth_path=/sbnvju-56/ 2024-03-04T17:06:20.311 DEBUG:boto:Method: PUT 2024-03-04T17:06:20.312 DEBUG:boto:Path: /sbnvju-56/ 2024-03-04T17:06:20.312 DEBUG:boto:Data: 2024-03-04T17:06:20.312 DEBUG:boto:Headers: {} 2024-03-04T17:06:20.312 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:06:20.312 DEBUG:boto:Port: 8000 2024-03-04T17:06:20.312 DEBUG:boto:Params: {} 2024-03-04T17:06:20.312 DEBUG:boto:Token: None 2024-03-04T17:06:20.312 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:20 GMT /sbnvju-56/ 2024-03-04T17:06:20.312 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pR8DDo/vFdsie+jgeYv0jg5XP2I= 2024-03-04T17:06:20.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pR8DDo/vFdsie+jgeYv0jg5XP2I=', 'Content-Length': '0'} 2024-03-04T17:06:20.350 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000afe47f9ac55bf56a-0065e5ff8c-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:20.350 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-57 2024-03-04T17:06:20.350 DEBUG:boto:path=/sbnvju-57/ 2024-03-04T17:06:20.351 DEBUG:boto:auth_path=/sbnvju-57/ 2024-03-04T17:06:20.351 DEBUG:boto:Method: PUT 2024-03-04T17:06:20.351 DEBUG:boto:Path: /sbnvju-57/ 2024-03-04T17:06:20.351 DEBUG:boto:Data: 2024-03-04T17:06:20.351 DEBUG:boto:Headers: {} 2024-03-04T17:06:20.351 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:06:20.351 DEBUG:boto:Port: 8001 2024-03-04T17:06:20.351 DEBUG:boto:Params: {} 2024-03-04T17:06:20.351 DEBUG:boto:Token: None 2024-03-04T17:06:20.351 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:06:20 GMT /sbnvju-57/ 2024-03-04T17:06:20.352 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5naGR1meLVAGdOMauEe94k94r5Y= 2024-03-04T17:06:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5naGR1meLVAGdOMauEe94k94r5Y=', 'Content-Length': '0'} 2024-03-04T17:06:20.381 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5b0cb2c7d1547eb-0065e5ff8c-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:06:20.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:20.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:20.382 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:20.434 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:20.434 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:20.485 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:06:20.505 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709571980.387781_747.1","last_update":"2024-03-04T17:06:20.387781Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:06:20.506 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:20.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:06:20.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:20.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:20.506 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:20.669 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:20.669 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:20.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:20.747+0000 7fd91a133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:20.748+0000 7fd91a133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:20.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:20Z"} 2024-03-04T17:06:20.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:20Z"} 2024-03-04T17:06:20.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:20.867 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:20.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:20.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T17:06:25.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:25.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:25.870 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:26.095 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:26.095 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:26.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:26.279+0000 7facb380f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:26.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:26.280+0000 7facb380f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:26.292 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:26Z"} 2024-03-04T17:06:26.292 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:26Z"} 2024-03-04T17:06:26.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:26.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:26.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:26.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:06:31.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:31.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:31.294 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:31.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:31.340 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:31.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:31.422+0000 7f2e72ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:31.423+0000 7f2e72ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:31.435 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:31Z"} 2024-03-04T17:06:31.436 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:31Z"} 2024-03-04T17:06:31.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:31.436 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:31.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:31.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T17:06:36.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:36.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:36.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:36.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:36.570+0000 7f766ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:36.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:36.571+0000 7f766ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:36.579 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:36Z"} 2024-03-04T17:06:36.579 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:36Z"} 2024-03-04T17:06:36.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:36.580 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:36.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:36.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T17:06:41.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:41.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:41.582 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:41.635 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:41.635 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:41.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:41.705+0000 7fe9d5616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:41.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:41.706+0000 7fe9d5616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:41.716 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:41Z"} 2024-03-04T17:06:41.716 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:41Z"} 2024-03-04T17:06:41.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:41.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:41.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:41.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570372.193667_464.1 2024-03-04T17:06:46.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:46.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:46.719 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:46.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:46.771 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:46.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:46.855+0000 7f2f1ac17800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:46.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:46.856+0000 7f2f1ac17800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:46.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:46Z"} 2024-03-04T17:06:46.867 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:46Z"} 2024-03-04T17:06:46.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:46.867 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:46.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:46.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T17:06:51.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:51.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:51.870 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:51.917 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:51.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:51.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:51.985+0000 7f5ff9da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:51.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:51.986+0000 7f5ff9da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:51.995 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:51Z"} 2024-03-04T17:06:51.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:51Z"} 2024-03-04T17:06:51.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:51.996 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:51.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:51.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T17:06:56.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:56.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:06:56.997 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:06:57.047 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:06:57.048 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:06:57.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:57.323+0000 7f7cf0213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:57.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:06:57.324+0000 7f7cf0213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:06:57.365 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:57Z"} 2024-03-04T17:06:57.365 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:57Z"} 2024-03-04T17:06:57.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:06:57.365 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:57.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:06:57.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.975603_447.1 2024-03-04T17:07:02.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:02.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:02.367 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:02.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:02.581 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:02.596+0000 7f71f6d2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:02.582 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:02.598+0000 7f71f6d2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:02.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:02Z"} 2024-03-04T17:07:02.630 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:02Z"} 2024-03-04T17:07:02.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:02.630 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:02.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:02.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T17:07:07.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:07.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:07.632 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:07.705 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:07.706 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:07.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:07.801+0000 7f7e7bb11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:07.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:07.802+0000 7f7e7bb11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:07.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:07Z"} 2024-03-04T17:07:07.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:07Z"} 2024-03-04T17:07:07.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:07.810 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:07.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:07.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.948368_440.1 2024-03-04T17:07:12.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:12.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:12.811 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:12.864 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:12.864 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:12.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:12.943+0000 7fca0a730800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:12.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:12.944+0000 7fca0a730800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:12.953 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:12Z"} 2024-03-04T17:07:12.954 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:12Z"} 2024-03-04T17:07:12.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:12.954 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:12.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:12.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:07:17.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:17.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:17.956 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:18.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:18.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:18.085+0000 7f8679e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:18.085+0000 7f8679e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:18.098 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:18Z"} 2024-03-04T17:07:18.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:18Z"} 2024-03-04T17:07:18.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:18.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:18.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:18.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T17:07:23.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:23.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:23.101 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:23.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:23.147 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:23.218+0000 7fc26d82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:23.219+0000 7fc26d82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:23.226 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:23Z"} 2024-03-04T17:07:23.226 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:23Z"} 2024-03-04T17:07:23.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:23.226 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:23.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:23.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T17:07:28.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:28.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:28.228 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:28.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:28.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:28.441 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:28.456+0000 7fe97ce20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:28.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:28.457+0000 7fe97ce20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:28.464 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:28Z"} 2024-03-04T17:07:28.464 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:28Z"} 2024-03-04T17:07:28.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:28.465 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:28.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:28.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T17:07:33.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:33.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:33.467 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:33.514 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:33.514 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:33.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:33.589+0000 7ff8f7030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:33.575 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:33.590+0000 7ff8f7030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:33.599 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:33Z"} 2024-03-04T17:07:33.600 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:33Z"} 2024-03-04T17:07:33.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:33.600 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:33.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:33.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:07:38.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:38.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:38.602 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:38.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:38.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:38.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:38.710+0000 7f37db326800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:38.696 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:38.711+0000 7f37db326800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:38.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:38Z"} 2024-03-04T17:07:38.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:38Z"} 2024-03-04T17:07:38.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:38.720 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:38.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:38.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.015683_453.1 2024-03-04T17:07:43.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:43.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:43.721 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:43.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:43.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:43.821 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:43.836+0000 7f6240d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:43.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:43.837+0000 7f6240d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:43.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:43Z"} 2024-03-04T17:07:43.848 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:43Z"} 2024-03-04T17:07:43.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:43.848 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:43.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:43.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T17:07:48.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:48.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:48.849 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:48.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:48.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:48.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:48.983+0000 7f427c53c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:48.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:48.984+0000 7f427c53c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:48.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:48Z"} 2024-03-04T17:07:48.994 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:48Z"} 2024-03-04T17:07:48.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:48.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:48.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:48.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.973308_446.1 2024-03-04T17:07:53.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:53.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:53.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:54.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:54.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:54.099 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:54.113+0000 7fbf14930800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:54.100 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:54.114+0000 7fbf14930800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:54.122 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:54Z"} 2024-03-04T17:07:54.122 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:54Z"} 2024-03-04T17:07:54.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:54.123 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:54.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:54.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:07:59.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:59.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:07:59.125 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:07:59.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:07:59.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:07:59.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:59.254+0000 7f1952c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:59.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:07:59.255+0000 7f1952c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:07:59.266 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:59Z"} 2024-03-04T17:07:59.266 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:59Z"} 2024-03-04T17:07:59.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:07:59.266 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:59.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:07:59.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:08:04.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:04.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:04.268 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:04.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:04.317 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:04.401+0000 7fe11b42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:04.401+0000 7fe11b42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:04.409 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:04Z"} 2024-03-04T17:08:04.409 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:04Z"} 2024-03-04T17:08:04.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:04.410 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:04.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:04.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T17:08:09.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:09.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:09.411 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:09.458 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:09.458 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:09.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:09.540+0000 7f8786812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:09.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:09.541+0000 7f8786812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:09.558 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:09Z"} 2024-03-04T17:08:09.558 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:09Z"} 2024-03-04T17:08:09.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:09.558 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:09.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:09.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.858235_434.1 2024-03-04T17:08:14.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:14.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:14.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:14.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:14.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:14.671 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:14.685+0000 7f50ffd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:14.672 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:14.686+0000 7f50ffd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:14.700 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:14Z"} 2024-03-04T17:08:14.700 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:14Z"} 2024-03-04T17:08:14.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:14.700 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:14.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:14.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T17:08:18.947 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:08:18.959+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:08:18.983 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:08:18.997+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:08:19.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:19.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:19.703 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:19.750 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:19.750 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:19.812 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:19.826+0000 7f2d39a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:19.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:19.827+0000 7f2d39a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:19.849 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:19Z"} 2024-03-04T17:08:19.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:19Z"} 2024-03-04T17:08:19.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:19.849 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:19.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:19.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:08:24.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:24.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:24.851 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:24.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:24.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:24.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:24.983+0000 7ff7cb224800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:24.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:24.984+0000 7ff7cb224800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:25.001 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:24Z"} 2024-03-04T17:08:25.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:24Z"} 2024-03-04T17:08:25.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:25.002 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:25.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:25.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T17:08:30.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:30.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:30.004 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:30.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:30.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:30.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:30.133+0000 7f9c72e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:30.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:30.133+0000 7f9c72e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:30.149 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:30Z"} 2024-03-04T17:08:30.149 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:30Z"} 2024-03-04T17:08:30.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:30.150 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:30.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:30.150 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.013079_452.1 2024-03-04T17:08:35.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:35.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:35.152 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:35.203 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:35.203 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:35.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:35.281+0000 7ff7bf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:35.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:35.282+0000 7ff7bf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:35.293 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:35Z"} 2024-03-04T17:08:35.293 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:35Z"} 2024-03-04T17:08:35.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:35.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:35.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:35.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T17:08:40.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:40.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:40.295 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:40.368 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:40.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:40.425 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:40.438+0000 7f43e15a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:40.426 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:40.439+0000 7f43e15a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:40.448 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:40Z"} 2024-03-04T17:08:40.448 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:40Z"} 2024-03-04T17:08:40.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:40.449 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:40.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:40.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:08:45.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:45.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:45.451 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:45.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:45.507 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:45.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:45.625+0000 7f4decf11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:45.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:45.631+0000 7f4decf11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:45.719 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:45Z"} 2024-03-04T17:08:45.719 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:45Z"} 2024-03-04T17:08:45.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:45.720 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:45.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:45.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T17:08:50.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:50.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:50.721 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:50.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:50.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:50.917+0000 7f8135933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:50.905 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:50.918+0000 7f8135933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:50.951 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:50Z"} 2024-03-04T17:08:50.951 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:50Z"} 2024-03-04T17:08:50.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:50.952 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:50.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:50.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:08:55.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:55.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:08:55.954 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:08:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:08:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:08:56.063 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:56.075+0000 7f1b93733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:56.063 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:08:56.076+0000 7f1b93733800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:08:56.089 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:56Z"} 2024-03-04T17:08:56.089 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:56Z"} 2024-03-04T17:08:56.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:08:56.089 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:56.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:08:56.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:09:01.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:01.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:01.092 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:01.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:01.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:01.196 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:01.209+0000 7fbf7f139800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:01.197 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:01.210+0000 7fbf7f139800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:01.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:01Z"} 2024-03-04T17:09:01.221 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:01Z"} 2024-03-04T17:09:01.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:01.222 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:01.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:01.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T17:09:06.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:06.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:06.223 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:06.331+0000 7f3b8a911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:06.332+0000 7f3b8a911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:06.341 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:06Z"} 2024-03-04T17:09:06.341 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:06Z"} 2024-03-04T17:09:06.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:06.341 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:06.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:06.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T17:09:11.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:11.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:11.343 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:11.505 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:11.505 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:11.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:11.580+0000 7fc7af02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:11.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:11.581+0000 7fc7af02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:11.595 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:11Z"} 2024-03-04T17:09:11.595 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:11Z"} 2024-03-04T17:09:11.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:11.596 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:11.596 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:11.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:09:16.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:16.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:16.597 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:16.643 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:16.643 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:16.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:16.715+0000 7f030a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:16.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:16.716+0000 7f030a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:16.725 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:16Z"} 2024-03-04T17:09:16.726 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:16Z"} 2024-03-04T17:09:16.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:16.726 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:16.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:16.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:09:21.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:21.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:21.728 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:21.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:21.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:21.875 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:21.887+0000 7ff37d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:21.876 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:21.888+0000 7ff37d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:21.901 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:21Z"} 2024-03-04T17:09:21.901 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:21Z"} 2024-03-04T17:09:21.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:21.902 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:21.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:21.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T17:09:26.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:26.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:26.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:26.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:26.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:27.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:27.015+0000 7f872e20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:27.004 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:27.016+0000 7f872e20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:27.026 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:27Z"} 2024-03-04T17:09:27.026 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:27Z"} 2024-03-04T17:09:27.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:27.027 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:27.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:27.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T17:09:32.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:32.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:32.029 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:32.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:32.241 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:32.317 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:32.329+0000 7f137c54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:32.318 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:32.330+0000 7f137c54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:32.345 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:32Z"} 2024-03-04T17:09:32.345 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:32Z"} 2024-03-04T17:09:32.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:32.346 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:32.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:32.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.934048_439.1 2024-03-04T17:09:37.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:37.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:37.348 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:37.397 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:37.397 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:37.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:37.477+0000 7f085bd30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:37.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:37.478+0000 7f085bd30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:37.490 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:37Z"} 2024-03-04T17:09:37.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:37Z"} 2024-03-04T17:09:37.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:37.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:37.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:37.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:09:42.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:42.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:42.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:42.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:42.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:42.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:42.617+0000 7f6620d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:42.607 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:42.618+0000 7f6620d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:42.633 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:42Z"} 2024-03-04T17:09:42.634 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:42Z"} 2024-03-04T17:09:42.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:42.634 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:42.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:42.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.994542_448.1 2024-03-04T17:09:47.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:47.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:47.637 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:47.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:47.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:47.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:47.746+0000 7fcd5c133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:47.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:47.747+0000 7fcd5c133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:47.759 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:47Z"} 2024-03-04T17:09:47.759 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:47Z"} 2024-03-04T17:09:47.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:47.759 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:47.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:47.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:09:52.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:52.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:52.761 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:52.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:52.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:52.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:52.875+0000 7f4cd9e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:52.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:52.876+0000 7f4cd9e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:52.887 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:52Z"} 2024-03-04T17:09:52.887 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:52Z"} 2024-03-04T17:09:52.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:52.888 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:52.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:52.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:09:57.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:57.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:09:57.890 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:09:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:09:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:09:57.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:58.010+0000 7fd01e616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:58.000 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:09:58.011+0000 7fd01e616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:09:58.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:58Z"} 2024-03-04T17:09:58.031 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:58Z"} 2024-03-04T17:09:58.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:09:58.032 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:58.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:09:58.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:10:03.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:03.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:03.033 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:03.165+0000 7f010fc67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:03.166+0000 7f010fc67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:03.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:03Z"} 2024-03-04T17:10:03.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:03Z"} 2024-03-04T17:10:03.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:03.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:03.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:03.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.013079_452.1 2024-03-04T17:10:08.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:08.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:08.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:08.238 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:08.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:08.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:08.301+0000 7f00fa133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:08.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:08.302+0000 7f00fa133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:08.315 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:08Z"} 2024-03-04T17:10:08.315 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:08Z"} 2024-03-04T17:10:08.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:08.315 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:08.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:08.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.973308_446.1 2024-03-04T17:10:13.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:13.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:13.317 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:13.365 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:13.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:13.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:13.429+0000 7fc8d6e2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:13.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:13.430+0000 7fc8d6e2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:13.441 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:13Z"} 2024-03-04T17:10:13.441 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:13Z"} 2024-03-04T17:10:13.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:13.442 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:13.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:13.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T17:10:18.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:18.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:18.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:18.493 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:18.493 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:18.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:18.571+0000 7f95e4e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:18.580 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:18.591+0000 7f95e4e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:18.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:18Z"} 2024-03-04T17:10:18.677 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:18Z"} 2024-03-04T17:10:18.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:18.677 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:18.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:18.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:10:23.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:23.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:23.679 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:23.750 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:23.751 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:23.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:23.909+0000 7f831988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:23.899 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:23.910+0000 7f831988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:23.921 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:23Z"} 2024-03-04T17:10:23.921 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:23Z"} 2024-03-04T17:10:23.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:23.922 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:23.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:23.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.948368_440.1 2024-03-04T17:10:28.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:28.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:28.924 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:29.030 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:29.040+0000 7f4a6be11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:29.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:29.041+0000 7f4a6be11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:29.052 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:29Z"} 2024-03-04T17:10:29.053 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:29Z"} 2024-03-04T17:10:29.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:29.053 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:29.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:29.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T17:10:34.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:34.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:34.055 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:34.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:34.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:34.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:34.175+0000 7f99755a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:34.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:34.176+0000 7f99755a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:34.190 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:34Z"} 2024-03-04T17:10:34.190 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:34Z"} 2024-03-04T17:10:34.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:34.191 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:34.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:34.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:10:39.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:39.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:39.193 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:39.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:39.302+0000 7f432308c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:39.302+0000 7f432308c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:39.314 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:39Z"} 2024-03-04T17:10:39.315 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:39Z"} 2024-03-04T17:10:39.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:39.315 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:39.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:39.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T17:10:44.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:44.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:44.316 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:44.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:44.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:44.527+0000 7fd653610800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:44.517 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:44.528+0000 7fd653610800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:44.542 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:44Z"} 2024-03-04T17:10:44.543 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:44Z"} 2024-03-04T17:10:44.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:44.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:44.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:44.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.997952_450.1 2024-03-04T17:10:49.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:49.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:49.544 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:49.592 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:49.592 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:49.653 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:49.664+0000 7f113c530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:49.654 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:49.665+0000 7f113c530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:49.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:49Z"} 2024-03-04T17:10:49.679 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:49Z"} 2024-03-04T17:10:49.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:49.680 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:49.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:49.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T17:10:54.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:54.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:54.682 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:54.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:54.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:54.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:54.810+0000 7f556f73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:54.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:54.811+0000 7f556f73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:54.828 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:54Z"} 2024-03-04T17:10:54.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:54Z"} 2024-03-04T17:10:54.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:54.829 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:54.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:54.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T17:10:59.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:59.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:10:59.831 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:10:59.882 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:10:59.882 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:10:59.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:59.944+0000 7f4fb7a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:59.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:10:59.945+0000 7f4fb7a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:10:59.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:59Z"} 2024-03-04T17:10:59.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:59Z"} 2024-03-04T17:10:59.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:10:59.958 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:59.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:10:59.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.972318_445.1 2024-03-04T17:11:04.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:04.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:04.960 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:05.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:05.083+0000 7ff3af2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:05.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:05.084+0000 7ff3af2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:05.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:05Z"} 2024-03-04T17:11:05.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:05Z"} 2024-03-04T17:11:05.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:05.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:05.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:05.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:11:10.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:10.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:10.101 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:10.151 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:10.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:10.215 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:10.224+0000 7f9e14e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:10.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:10.225+0000 7f9e14e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:10.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:10Z"} 2024-03-04T17:11:10.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:10Z"} 2024-03-04T17:11:10.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:10.246 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:10.247 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:10.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:11:15.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:15.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:15.248 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:15.300 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:15.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:15.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:15.364+0000 7fbd7a20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:15.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:15.365+0000 7fbd7a20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:15.378 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:15Z"} 2024-03-04T17:11:15.378 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:15Z"} 2024-03-04T17:11:15.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:15.378 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:15.379 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:15.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:11:20.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:20.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:20.380 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:20.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:20.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:20.496+0000 7f50732c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:20.497+0000 7f50732c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:20.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:20Z"} 2024-03-04T17:11:20.510 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:20Z"} 2024-03-04T17:11:20.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:20.510 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:20.510 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:20.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.927093_438.1 2024-03-04T17:11:25.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:25.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:25.512 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:25.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:25.560 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:25.630 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:25.639+0000 7f5008011800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:25.631 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:25.640+0000 7f5008011800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:25.652 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:25Z"} 2024-03-04T17:11:25.652 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:25Z"} 2024-03-04T17:11:25.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:25.653 DEBUG:rgw_multi.tests:log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:25.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:25.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T17:11:30.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-03-04T17:11:30.661 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-58 2024-03-04T17:11:30.661 DEBUG:boto:path=/sbnvju-58/ 2024-03-04T17:11:30.661 DEBUG:boto:auth_path=/sbnvju-58/ 2024-03-04T17:11:30.661 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.661 DEBUG:boto:Path: /sbnvju-58/ 2024-03-04T17:11:30.661 DEBUG:boto:Data: 2024-03-04T17:11:30.661 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.662 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.662 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.662 DEBUG:boto:Params: {} 2024-03-04T17:11:30.662 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:11:30.662 DEBUG:boto:Token: None 2024-03-04T17:11:30.663 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/ 2024-03-04T17:11:30.663 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:MxGvxYwl/d+ex3qbaQvROa3nuhM= 2024-03-04T17:11:30.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MxGvxYwl/d+ex3qbaQvROa3nuhM=', 'Content-Length': '0'} 2024-03-04T17:11:30.687 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd5297ecfda2077c-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.687 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-59 2024-03-04T17:11:30.687 DEBUG:boto:path=/sbnvju-59/ 2024-03-04T17:11:30.687 DEBUG:boto:auth_path=/sbnvju-59/ 2024-03-04T17:11:30.687 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.687 DEBUG:boto:Path: /sbnvju-59/ 2024-03-04T17:11:30.687 DEBUG:boto:Data: 2024-03-04T17:11:30.687 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.687 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.688 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.688 DEBUG:boto:Params: {} 2024-03-04T17:11:30.688 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:11:30.688 DEBUG:boto:Token: None 2024-03-04T17:11:30.688 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/ 2024-03-04T17:11:30.688 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iDhR2VwJaiWslRAUgAx4VZuQSoc= 2024-03-04T17:11:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDhR2VwJaiWslRAUgAx4VZuQSoc=', 'Content-Length': '0'} 2024-03-04T17:11:30.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1392431780d9460-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.727 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-60 2024-03-04T17:11:30.727 DEBUG:boto:path=/sbnvju-60/ 2024-03-04T17:11:30.727 DEBUG:boto:auth_path=/sbnvju-60/ 2024-03-04T17:11:30.727 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.727 DEBUG:boto:Path: /sbnvju-60/ 2024-03-04T17:11:30.727 DEBUG:boto:Data: 2024-03-04T17:11:30.727 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.727 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:30.727 DEBUG:boto:Port: 8001 2024-03-04T17:11:30.728 DEBUG:boto:Params: {} 2024-03-04T17:11:30.728 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:11:30.728 DEBUG:boto:Token: None 2024-03-04T17:11:30.728 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-60/ 2024-03-04T17:11:30.728 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Bc/Bqz2JExG0iAANJjSQt147Pyg= 2024-03-04T17:11:30.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Bc/Bqz2JExG0iAANJjSQt147Pyg=', 'Content-Length': '0'} 2024-03-04T17:11:30.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000846a5fe8b5d8134-0065e600c2-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.760 DEBUG:boto:path=/sbnvju-58/ 2024-03-04T17:11:30.760 DEBUG:boto:auth_path=/sbnvju-58/ 2024-03-04T17:11:30.760 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.760 DEBUG:boto:Path: /sbnvju-58/ 2024-03-04T17:11:30.760 DEBUG:boto:Data: 2024-03-04T17:11:30.760 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.761 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.761 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.761 DEBUG:boto:Params: {} 2024-03-04T17:11:30.761 DEBUG:boto:Token: None 2024-03-04T17:11:30.761 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/ 2024-03-04T17:11:30.761 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T17:11:30.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T17:11:30.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091ace23e93c1c288-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.764 DEBUG:boto:path=/sbnvju-58/obj1 2024-03-04T17:11:30.764 DEBUG:boto:auth_path=/sbnvju-58/obj1 2024-03-04T17:11:30.765 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.765 DEBUG:boto:Path: /sbnvju-58/obj1 2024-03-04T17:11:30.765 DEBUG:boto:Data: 2024-03-04T17:11:30.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.765 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.765 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.765 DEBUG:boto:Params: {} 2024-03-04T17:11:30.765 DEBUG:boto:Token: None 2024-03-04T17:11:30.766 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/obj1 2024-03-04T17:11:30.766 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:m/tUU+merGqraCwlOLvyonrIsYo= 2024-03-04T17:11:30.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m/tUU+merGqraCwlOLvyonrIsYo='} 2024-03-04T17:11:30.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058a331f2a1bd2ac2-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.777 DEBUG:boto:path=/sbnvju-58/ 2024-03-04T17:11:30.777 DEBUG:boto:auth_path=/sbnvju-58/ 2024-03-04T17:11:30.778 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.778 DEBUG:boto:Path: /sbnvju-58/ 2024-03-04T17:11:30.778 DEBUG:boto:Data: 2024-03-04T17:11:30.778 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.778 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.778 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.778 DEBUG:boto:Params: {} 2024-03-04T17:11:30.778 DEBUG:boto:Token: None 2024-03-04T17:11:30.778 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/ 2024-03-04T17:11:30.778 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T17:11:30.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T17:11:30.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000792a0a1509516d98-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.784 DEBUG:boto:path=/sbnvju-58/obj2 2024-03-04T17:11:30.784 DEBUG:boto:auth_path=/sbnvju-58/obj2 2024-03-04T17:11:30.784 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.784 DEBUG:boto:Path: /sbnvju-58/obj2 2024-03-04T17:11:30.784 DEBUG:boto:Data: 2024-03-04T17:11:30.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.784 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.785 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.785 DEBUG:boto:Params: {} 2024-03-04T17:11:30.785 DEBUG:boto:Token: None 2024-03-04T17:11:30.785 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/obj2 2024-03-04T17:11:30.785 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:NYgGjUIDTKi2Sr8U2NQaojJSrTI= 2024-03-04T17:11:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NYgGjUIDTKi2Sr8U2NQaojJSrTI='} 2024-03-04T17:11:30.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cba7c552b87ba2bb-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.792 DEBUG:boto:path=/sbnvju-58/ 2024-03-04T17:11:30.792 DEBUG:boto:auth_path=/sbnvju-58/ 2024-03-04T17:11:30.792 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.792 DEBUG:boto:Path: /sbnvju-58/ 2024-03-04T17:11:30.792 DEBUG:boto:Data: 2024-03-04T17:11:30.792 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.792 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.793 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.793 DEBUG:boto:Params: {} 2024-03-04T17:11:30.793 DEBUG:boto:Token: None 2024-03-04T17:11:30.793 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/ 2024-03-04T17:11:30.793 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T17:11:30.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T17:11:30.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002358189e4c109f38-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.796 DEBUG:boto:path=/sbnvju-58/obj3 2024-03-04T17:11:30.796 DEBUG:boto:auth_path=/sbnvju-58/obj3 2024-03-04T17:11:30.796 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.796 DEBUG:boto:Path: /sbnvju-58/obj3 2024-03-04T17:11:30.796 DEBUG:boto:Data: 2024-03-04T17:11:30.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.796 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.796 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.796 DEBUG:boto:Params: {} 2024-03-04T17:11:30.796 DEBUG:boto:Token: None 2024-03-04T17:11:30.797 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/obj3 2024-03-04T17:11:30.797 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s4tddDViq/IOPJofDBOHfdMT05g= 2024-03-04T17:11:30.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s4tddDViq/IOPJofDBOHfdMT05g='} 2024-03-04T17:11:30.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008772bb9a5d2c5efd-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.840 DEBUG:boto:path=/sbnvju-58/ 2024-03-04T17:11:30.840 DEBUG:boto:auth_path=/sbnvju-58/ 2024-03-04T17:11:30.841 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.841 DEBUG:boto:Path: /sbnvju-58/ 2024-03-04T17:11:30.841 DEBUG:boto:Data: 2024-03-04T17:11:30.841 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.841 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.841 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.841 DEBUG:boto:Params: {} 2024-03-04T17:11:30.841 DEBUG:boto:Token: None 2024-03-04T17:11:30.841 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/ 2024-03-04T17:11:30.842 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T17:11:30.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T17:11:30.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f6ea4651459d63-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.847 DEBUG:boto:path=/sbnvju-58/obj4 2024-03-04T17:11:30.847 DEBUG:boto:auth_path=/sbnvju-58/obj4 2024-03-04T17:11:30.847 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.847 DEBUG:boto:Path: /sbnvju-58/obj4 2024-03-04T17:11:30.847 DEBUG:boto:Data: 2024-03-04T17:11:30.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.847 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:11:30.847 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.847 DEBUG:boto:Params: {} 2024-03-04T17:11:30.848 DEBUG:boto:Token: None 2024-03-04T17:11:30.848 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-58/obj4 2024-03-04T17:11:30.848 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2R3e9VN6f2eYp9g+pxK30DzWe8o= 2024-03-04T17:11:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2R3e9VN6f2eYp9g+pxK30DzWe8o='} 2024-03-04T17:11:30.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c68d8dfed497a337-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.853 DEBUG:boto:path=/sbnvju-59/ 2024-03-04T17:11:30.854 DEBUG:boto:auth_path=/sbnvju-59/ 2024-03-04T17:11:30.854 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.854 DEBUG:boto:Path: /sbnvju-59/ 2024-03-04T17:11:30.854 DEBUG:boto:Data: 2024-03-04T17:11:30.854 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.854 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.854 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.854 DEBUG:boto:Params: {} 2024-03-04T17:11:30.854 DEBUG:boto:Token: None 2024-03-04T17:11:30.854 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/ 2024-03-04T17:11:30.855 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T17:11:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T17:11:30.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008095ff1cece9a32c-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.859 DEBUG:boto:path=/sbnvju-59/obj1 2024-03-04T17:11:30.859 DEBUG:boto:auth_path=/sbnvju-59/obj1 2024-03-04T17:11:30.859 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.859 DEBUG:boto:Path: /sbnvju-59/obj1 2024-03-04T17:11:30.859 DEBUG:boto:Data: 2024-03-04T17:11:30.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.859 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.859 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.859 DEBUG:boto:Params: {} 2024-03-04T17:11:30.859 DEBUG:boto:Token: None 2024-03-04T17:11:30.860 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/obj1 2024-03-04T17:11:30.860 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QoVRB8IhsWUJ7G2xA7i/Z9k5Y/Y= 2024-03-04T17:11:30.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QoVRB8IhsWUJ7G2xA7i/Z9k5Y/Y='} 2024-03-04T17:11:30.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000768e9440bb15dfa4-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.878 DEBUG:boto:path=/sbnvju-59/ 2024-03-04T17:11:30.878 DEBUG:boto:auth_path=/sbnvju-59/ 2024-03-04T17:11:30.878 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.878 DEBUG:boto:Path: /sbnvju-59/ 2024-03-04T17:11:30.878 DEBUG:boto:Data: 2024-03-04T17:11:30.879 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.879 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.879 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.879 DEBUG:boto:Params: {} 2024-03-04T17:11:30.879 DEBUG:boto:Token: None 2024-03-04T17:11:30.879 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/ 2024-03-04T17:11:30.879 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T17:11:30.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T17:11:30.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000571f24dc1ede9b8d-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.881 DEBUG:boto:path=/sbnvju-59/obj2 2024-03-04T17:11:30.881 DEBUG:boto:auth_path=/sbnvju-59/obj2 2024-03-04T17:11:30.882 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.882 DEBUG:boto:Path: /sbnvju-59/obj2 2024-03-04T17:11:30.882 DEBUG:boto:Data: 2024-03-04T17:11:30.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.882 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.882 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.882 DEBUG:boto:Params: {} 2024-03-04T17:11:30.882 DEBUG:boto:Token: None 2024-03-04T17:11:30.882 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/obj2 2024-03-04T17:11:30.882 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:sXTH6NIbYnmpKhF8rwoYA7vFJHk= 2024-03-04T17:11:30.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sXTH6NIbYnmpKhF8rwoYA7vFJHk='} 2024-03-04T17:11:30.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2bb71f9f5849460-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.924 DEBUG:boto:path=/sbnvju-59/ 2024-03-04T17:11:30.924 DEBUG:boto:auth_path=/sbnvju-59/ 2024-03-04T17:11:30.925 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.925 DEBUG:boto:Path: /sbnvju-59/ 2024-03-04T17:11:30.925 DEBUG:boto:Data: 2024-03-04T17:11:30.925 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.925 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.925 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.925 DEBUG:boto:Params: {} 2024-03-04T17:11:30.925 DEBUG:boto:Token: None 2024-03-04T17:11:30.925 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/ 2024-03-04T17:11:30.925 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T17:11:30.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T17:11:30.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b6469d48d2f1524-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.928 DEBUG:boto:path=/sbnvju-59/obj3 2024-03-04T17:11:30.928 DEBUG:boto:auth_path=/sbnvju-59/obj3 2024-03-04T17:11:30.929 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.929 DEBUG:boto:Path: /sbnvju-59/obj3 2024-03-04T17:11:30.929 DEBUG:boto:Data: 2024-03-04T17:11:30.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.929 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.929 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.929 DEBUG:boto:Params: {} 2024-03-04T17:11:30.929 DEBUG:boto:Token: None 2024-03-04T17:11:30.929 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/obj3 2024-03-04T17:11:30.929 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:WYxUANy1Z6NAoPrzvbUk1/y6Aeo= 2024-03-04T17:11:30.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WYxUANy1Z6NAoPrzvbUk1/y6Aeo='} 2024-03-04T17:11:30.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c91f5c552f56a22-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.946 DEBUG:boto:path=/sbnvju-59/ 2024-03-04T17:11:30.947 DEBUG:boto:auth_path=/sbnvju-59/ 2024-03-04T17:11:30.947 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.947 DEBUG:boto:Path: /sbnvju-59/ 2024-03-04T17:11:30.947 DEBUG:boto:Data: 2024-03-04T17:11:30.947 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.947 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.947 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.947 DEBUG:boto:Params: {} 2024-03-04T17:11:30.947 DEBUG:boto:Token: None 2024-03-04T17:11:30.947 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/ 2024-03-04T17:11:30.948 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T17:11:30.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T17:11:30.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b15916e85a7c5bcf-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.951 DEBUG:boto:path=/sbnvju-59/obj4 2024-03-04T17:11:30.951 DEBUG:boto:auth_path=/sbnvju-59/obj4 2024-03-04T17:11:30.951 DEBUG:boto:Method: PUT 2024-03-04T17:11:30.951 DEBUG:boto:Path: /sbnvju-59/obj4 2024-03-04T17:11:30.951 DEBUG:boto:Data: 2024-03-04T17:11:30.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:30.951 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:11:30.951 DEBUG:boto:Port: 8000 2024-03-04T17:11:30.951 DEBUG:boto:Params: {} 2024-03-04T17:11:30.952 DEBUG:boto:Token: None 2024-03-04T17:11:30.952 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-59/obj4 2024-03-04T17:11:30.952 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+izsiITcpqkbnvB1bzHJuoHX4Yk= 2024-03-04T17:11:30.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+izsiITcpqkbnvB1bzHJuoHX4Yk='} 2024-03-04T17:11:30.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6fb78e393b7615-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:30.996 DEBUG:boto:path=/sbnvju-60/ 2024-03-04T17:11:30.996 DEBUG:boto:auth_path=/sbnvju-60/ 2024-03-04T17:11:30.997 DEBUG:boto:Method: HEAD 2024-03-04T17:11:30.997 DEBUG:boto:Path: /sbnvju-60/ 2024-03-04T17:11:30.997 DEBUG:boto:Data: 2024-03-04T17:11:30.997 DEBUG:boto:Headers: {} 2024-03-04T17:11:30.997 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:30.997 DEBUG:boto:Port: 8001 2024-03-04T17:11:30.997 DEBUG:boto:Params: {} 2024-03-04T17:11:30.997 DEBUG:boto:Token: None 2024-03-04T17:11:30.997 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:30 GMT /sbnvju-60/ 2024-03-04T17:11:30.997 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:t6VGvEsxsE+0aHfCU6aCrQkGFuI= 2024-03-04T17:11:30.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t6VGvEsxsE+0aHfCU6aCrQkGFuI=', 'Content-Length': '0'} 2024-03-04T17:11:31.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073df105823d98088-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.000 DEBUG:boto:path=/sbnvju-60/obj1 2024-03-04T17:11:31.000 DEBUG:boto:auth_path=/sbnvju-60/obj1 2024-03-04T17:11:31.001 DEBUG:boto:Method: PUT 2024-03-04T17:11:31.001 DEBUG:boto:Path: /sbnvju-60/obj1 2024-03-04T17:11:31.001 DEBUG:boto:Data: 2024-03-04T17:11:31.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:31.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.001 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.001 DEBUG:boto:Params: {} 2024-03-04T17:11:31.001 DEBUG:boto:Token: None 2024-03-04T17:11:31.001 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/obj1 2024-03-04T17:11:31.001 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TvN9T2rC5Lp1aXLJhNnmibG5bmI= 2024-03-04T17:11:31.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TvN9T2rC5Lp1aXLJhNnmibG5bmI='} 2024-03-04T17:11:31.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b966b99d2c6b47-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.009 DEBUG:boto:path=/sbnvju-60/ 2024-03-04T17:11:31.009 DEBUG:boto:auth_path=/sbnvju-60/ 2024-03-04T17:11:31.009 DEBUG:boto:Method: HEAD 2024-03-04T17:11:31.009 DEBUG:boto:Path: /sbnvju-60/ 2024-03-04T17:11:31.009 DEBUG:boto:Data: 2024-03-04T17:11:31.009 DEBUG:boto:Headers: {} 2024-03-04T17:11:31.009 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.009 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.009 DEBUG:boto:Params: {} 2024-03-04T17:11:31.009 DEBUG:boto:Token: None 2024-03-04T17:11:31.009 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/ 2024-03-04T17:11:31.010 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T17:11:31.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T17:11:31.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dbb5830764abcda-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.012 DEBUG:boto:path=/sbnvju-60/obj2 2024-03-04T17:11:31.012 DEBUG:boto:auth_path=/sbnvju-60/obj2 2024-03-04T17:11:31.013 DEBUG:boto:Method: PUT 2024-03-04T17:11:31.013 DEBUG:boto:Path: /sbnvju-60/obj2 2024-03-04T17:11:31.013 DEBUG:boto:Data: 2024-03-04T17:11:31.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:31.013 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.013 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.013 DEBUG:boto:Params: {} 2024-03-04T17:11:31.013 DEBUG:boto:Token: None 2024-03-04T17:11:31.013 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/obj2 2024-03-04T17:11:31.014 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:aokVgM9o3MkCPWDZvTfJNlNV0zM= 2024-03-04T17:11:31.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aokVgM9o3MkCPWDZvTfJNlNV0zM='} 2024-03-04T17:11:31.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030e0f65a3f183b18-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.019 DEBUG:boto:path=/sbnvju-60/ 2024-03-04T17:11:31.019 DEBUG:boto:auth_path=/sbnvju-60/ 2024-03-04T17:11:31.019 DEBUG:boto:Method: HEAD 2024-03-04T17:11:31.019 DEBUG:boto:Path: /sbnvju-60/ 2024-03-04T17:11:31.019 DEBUG:boto:Data: 2024-03-04T17:11:31.019 DEBUG:boto:Headers: {} 2024-03-04T17:11:31.020 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.020 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.020 DEBUG:boto:Params: {} 2024-03-04T17:11:31.020 DEBUG:boto:Token: None 2024-03-04T17:11:31.020 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/ 2024-03-04T17:11:31.020 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T17:11:31.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T17:11:31.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693353b7acaecdb1-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.023 DEBUG:boto:path=/sbnvju-60/obj3 2024-03-04T17:11:31.023 DEBUG:boto:auth_path=/sbnvju-60/obj3 2024-03-04T17:11:31.023 DEBUG:boto:Method: PUT 2024-03-04T17:11:31.023 DEBUG:boto:Path: /sbnvju-60/obj3 2024-03-04T17:11:31.023 DEBUG:boto:Data: 2024-03-04T17:11:31.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:31.023 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.023 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.023 DEBUG:boto:Params: {} 2024-03-04T17:11:31.023 DEBUG:boto:Token: None 2024-03-04T17:11:31.024 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/obj3 2024-03-04T17:11:31.024 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:AIYYSPbLNTgTXlBJoMzi9tdEAL4= 2024-03-04T17:11:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AIYYSPbLNTgTXlBJoMzi9tdEAL4='} 2024-03-04T17:11:31.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a858cf4c1bcdd25-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.068 DEBUG:boto:path=/sbnvju-60/ 2024-03-04T17:11:31.068 DEBUG:boto:auth_path=/sbnvju-60/ 2024-03-04T17:11:31.069 DEBUG:boto:Method: HEAD 2024-03-04T17:11:31.069 DEBUG:boto:Path: /sbnvju-60/ 2024-03-04T17:11:31.069 DEBUG:boto:Data: 2024-03-04T17:11:31.069 DEBUG:boto:Headers: {} 2024-03-04T17:11:31.069 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.069 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.069 DEBUG:boto:Params: {} 2024-03-04T17:11:31.069 DEBUG:boto:Token: None 2024-03-04T17:11:31.069 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/ 2024-03-04T17:11:31.069 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T17:11:31.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T17:11:31.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d3e0fc9963ad62-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.072 DEBUG:boto:path=/sbnvju-60/obj4 2024-03-04T17:11:31.072 DEBUG:boto:auth_path=/sbnvju-60/obj4 2024-03-04T17:11:31.072 DEBUG:boto:Method: PUT 2024-03-04T17:11:31.072 DEBUG:boto:Path: /sbnvju-60/obj4 2024-03-04T17:11:31.072 DEBUG:boto:Data: 2024-03-04T17:11:31.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:11:31.072 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:11:31.072 DEBUG:boto:Port: 8001 2024-03-04T17:11:31.072 DEBUG:boto:Params: {} 2024-03-04T17:11:31.073 DEBUG:boto:Token: None 2024-03-04T17:11:31.073 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:11:31 GMT /sbnvju-60/obj4 2024-03-04T17:11:31.073 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:s0HjZmJtTgygGQ82Kgdb0fseULo= 2024-03-04T17:11:31.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s0HjZmJtTgygGQ82Kgdb0fseULo='} 2024-03-04T17:11:31.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec00a4b90c7e0cb0-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:11:31.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:31.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:31.079 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:31.127 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:31.127 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:11:31.192 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709572290.757934_773.1","last_update":"2024-03-04T17:11:30.757934Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:11:31.192 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:31.193 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:11:31.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:31.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:31.194 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:31.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:31.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:31.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:31.309+0000 7f52be309800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:31.301 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:31.310+0000 7f52be309800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:31.326 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:31Z"} 2024-03-04T17:11:31.326 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:31Z"} 2024-03-04T17:11:31.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:31.327 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:31.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:31.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.954325_443.1 2024-03-04T17:11:36.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:36.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:36.332 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:36.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:36.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:36.514+0000 7f765d430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:36.515+0000 7f765d430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:36.536 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:36Z"} 2024-03-04T17:11:36.536 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:36Z"} 2024-03-04T17:11:36.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:36.536 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:36.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:36.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T17:11:41.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:41.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:41.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:41.587 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:41.588 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:41.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:41.654+0000 7f58eec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:41.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:41.655+0000 7f58eec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:41.668 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:41Z"} 2024-03-04T17:11:41.668 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:41Z"} 2024-03-04T17:11:41.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:41.669 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:41.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:41.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570988.612016_560.1 2024-03-04T17:11:46.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:46.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:46.671 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:46.719 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:46.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:46.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:46.796+0000 7f478a32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:46.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:46.797+0000 7f478a32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:46.821 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:46Z"} 2024-03-04T17:11:46.822 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:46Z"} 2024-03-04T17:11:46.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:46.822 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:46.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:46.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T17:11:51.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:51.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:51.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:51.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:51.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:51.937 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:51.946+0000 7f1ab5faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:51.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:51.947+0000 7f1ab5faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:51.961 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:51Z"} 2024-03-04T17:11:51.961 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:51Z"} 2024-03-04T17:11:51.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:51.961 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:51.962 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:51.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.975603_447.1 2024-03-04T17:11:56.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:56.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:11:56.964 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:11:57.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:11:57.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:11:57.068 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:57.077+0000 7f1405a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:11:57.077+0000 7f1405a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:11:57.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:57Z"} 2024-03-04T17:11:57.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:57Z"} 2024-03-04T17:11:57.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:11:57.100 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:57.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:11:57.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.994542_448.1 2024-03-04T17:12:02.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:02.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:02.102 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:02.148 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:02.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:02.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:02.223+0000 7f5cc3f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:02.215 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:02.224+0000 7f5cc3f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:02.237 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:02Z"} 2024-03-04T17:12:02.237 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:02Z"} 2024-03-04T17:12:02.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:02.238 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:02.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:02.238 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.860903_435.1 2024-03-04T17:12:07.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:07.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:07.239 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:07.286 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:07.286 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:07.368+0000 7fafd2624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:07.369+0000 7fafd2624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:07.390 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:07Z"} 2024-03-04T17:12:07.390 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:07Z"} 2024-03-04T17:12:07.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:07.390 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:07.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:07.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:12:12.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:12.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:12.392 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:12.439 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:12.439 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:12.504 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:12.512+0000 7f1ccb12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:12.504 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:12.513+0000 7f1ccb12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:12.534 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:12Z"} 2024-03-04T17:12:12.535 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:12Z"} 2024-03-04T17:12:12.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:12.535 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:12.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:12.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T17:12:17.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:17.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:17.537 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:17.584 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:17.584 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:17.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:17.664+0000 7ff07c018800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:17.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:17.665+0000 7ff07c018800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:17.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:17Z"} 2024-03-04T17:12:17.679 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:17Z"} 2024-03-04T17:12:17.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:17.679 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:17.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:17.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:12:22.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:22.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:22.681 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:22.734 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:22.734 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:22.819+0000 7f2ad7e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:22.820+0000 7f2ad7e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:22Z"} 2024-03-04T17:12:22.842 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:22Z"} 2024-03-04T17:12:22.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:22.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:22.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:22.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.870879_436.1 2024-03-04T17:12:27.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:27.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:27.844 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:27.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:27.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:27.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:27.957+0000 7f5275939800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:27.958+0000 7f5275939800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:27.974 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:27Z"} 2024-03-04T17:12:27.974 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:27Z"} 2024-03-04T17:12:27.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:27.974 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:27.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:27.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T17:12:32.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:32.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:32.976 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:33.023 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:33.024 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:33.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:33.081+0000 7fe79bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:33.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:33.082+0000 7fe79bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:33.106 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:33Z"} 2024-03-04T17:12:33.106 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:33Z"} 2024-03-04T17:12:33.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:33.107 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:33.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:33.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T17:12:38.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:38.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:38.108 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:38.158 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:38.158 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:38.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:38.231+0000 7f7209015800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:38.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:38.232+0000 7f7209015800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:38.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:38Z"} 2024-03-04T17:12:38.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:38Z"} 2024-03-04T17:12:38.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:38.249 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:38.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:38.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T17:12:43.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:43.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:43.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:43.300 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:43.300 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:43.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:43.378+0000 7f4ec7416800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:43.370 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:43.379+0000 7f4ec7416800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:43.398 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:43Z"} 2024-03-04T17:12:43.398 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:43Z"} 2024-03-04T17:12:43.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:43.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:43.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:43.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:12:48.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:48.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:48.400 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:48.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:48.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:48.511 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:48.519+0000 7f3c86e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:48.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:48.520+0000 7f3c86e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:48.542 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:48Z"} 2024-03-04T17:12:48.542 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:48Z"} 2024-03-04T17:12:48.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:48.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:48.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:48.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:12:53.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:53.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:53.545 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:53.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:53.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:53.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:53.672+0000 7f7ef1444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:53.666 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:53.673+0000 7f7ef1444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:53.692 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:53Z"} 2024-03-04T17:12:53.692 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:53Z"} 2024-03-04T17:12:53.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:53.692 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:53.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:53.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:12:58.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:58.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:12:58.694 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:12:58.747 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:12:58.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:12:58.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:58.817+0000 7f0128a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:58.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:12:58.818+0000 7f0128a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:12:58.834 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:58Z"} 2024-03-04T17:12:58.835 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:58Z"} 2024-03-04T17:12:58.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:12:58.835 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:58.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:12:58.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:13:03.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:03.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:03.837 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:03.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:03.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:04.046 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:04.053+0000 7ffa6b239800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:04.047 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:04.054+0000 7ffa6b239800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:04.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:04Z"} 2024-03-04T17:13:04.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:04Z"} 2024-03-04T17:13:04.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:04.100 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:04.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:04.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:13:09.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:09.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:09.102 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:09.170 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:09.171 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:09.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:09.287+0000 7fec9034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:09.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:09.288+0000 7fec9034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:09.309 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:09Z"} 2024-03-04T17:13:09.309 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:09Z"} 2024-03-04T17:13:09.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:09.309 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:09.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:09.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.870879_436.1 2024-03-04T17:13:14.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:14.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:14.311 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:14.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:14.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:14.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:14.430+0000 7f9d32639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:14.424 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:14.431+0000 7f9d32639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:14Z"} 2024-03-04T17:13:14.447 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:14Z"} 2024-03-04T17:13:14.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:14.448 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:14.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:14.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.973308_446.1 2024-03-04T17:13:19.006 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:13:19.013+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:13:19.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:19.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:19.449 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:19.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:19.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:19.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:19.605+0000 7fcc3ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:19.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:19.606+0000 7fcc3ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:19.623 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:19Z"} 2024-03-04T17:13:19.623 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:19Z"} 2024-03-04T17:13:19.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:19.623 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:19.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:19.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:13:24.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:24.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:24.625 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:24.670 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:24.671 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:24.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:24.727+0000 7ff96a63a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:24.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:24.728+0000 7ff96a63a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:24.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:24Z"} 2024-03-04T17:13:24.743 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:24Z"} 2024-03-04T17:13:24.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:24.743 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:24.743 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:24.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.954325_443.1 2024-03-04T17:13:29.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:29.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:29.745 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:29.791 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:29.791 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:29.849 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:29.856+0000 7f69734c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:29.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:29.857+0000 7f69734c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:29.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:29Z"} 2024-03-04T17:13:29.877 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:29Z"} 2024-03-04T17:13:29.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:29.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:29.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:29.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:13:34.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:34.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:34.879 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:34.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:34.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:35.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:35.013+0000 7f6a9cc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:35.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:35.014+0000 7f6a9cc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:35.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:35Z"} 2024-03-04T17:13:35.040 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:35Z"} 2024-03-04T17:13:35.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:35.040 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:35.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:35.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:13:40.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:40.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:40.042 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:40.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:40.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:40.148+0000 7f4324133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:40.141 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:40.149+0000 7f4324133800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:40.164 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.196251_465.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.196251Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:40Z"} 2024-03-04T17:13:40.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.196251_465.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.196251Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:40Z"} 2024-03-04T17:13:40.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.196251_465.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.196251Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:40.165 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:40.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.196251_465.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:40.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.196251_465.1 2024-03-04T17:13:45.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:45.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:45.168 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:45.215 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:45.215 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:45.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:45.289+0000 7f18ebec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:45.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:45.290+0000 7f18ebec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:45.309 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:45Z"} 2024-03-04T17:13:45.309 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:45Z"} 2024-03-04T17:13:45.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:45.310 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:45.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:45.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T17:13:50.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:50.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:50.312 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:50.358 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:50.358 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:50.408 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:50.415+0000 7fbb5b244800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:50.409 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:50.416+0000 7fbb5b244800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:50.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:50Z"} 2024-03-04T17:13:50.442 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:50Z"} 2024-03-04T17:13:50.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:50.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:50.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:50.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T17:13:55.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:55.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:13:55.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:13:55.491 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:13:55.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:13:55.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:55.548+0000 7f7e8614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:55.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:13:55.549+0000 7f7e8614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:13:55.563 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:55Z"} 2024-03-04T17:13:55.564 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:55Z"} 2024-03-04T17:13:55.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:13:55.564 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:55.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:13:55.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.950988_441.1 2024-03-04T17:13:58.996 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:13:59.002+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:14:00.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:00.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:00.566 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:00.611 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:00.612 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:00.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:00.668+0000 7f6f9fe12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:00.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:00.669+0000 7f6f9fe12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:00.693 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:00Z"} 2024-03-04T17:14:00.693 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:00Z"} 2024-03-04T17:14:00.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:00.694 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:00.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:00.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:14:05.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:05.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:05.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:05.747 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:05.747 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:05.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:05.822+0000 7fd7171af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:05.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:05.823+0000 7fd7171af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:05.839 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:05Z"} 2024-03-04T17:14:05.840 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:05Z"} 2024-03-04T17:14:05.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:05.840 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:05.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:05.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:14:10.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:10.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:10.842 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:10.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:10.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:10.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:10.958+0000 7fd17a329800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:10.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:10.959+0000 7fd17a329800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:10.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:10Z"} 2024-03-04T17:14:10.978 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:10Z"} 2024-03-04T17:14:10.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:10.979 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:10.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:10.979 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:14:15.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:15.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:15.982 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:16.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:16.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:16.087 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:16.094+0000 7f5326f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:16.088 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:16.095+0000 7f5326f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:16.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:16Z"} 2024-03-04T17:14:16.113 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:16Z"} 2024-03-04T17:14:16.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:16.114 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:16.114 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:16.114 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T17:14:21.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:21.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:21.115 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:21.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:21.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:21.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:21.225+0000 7f1fdea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:21.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:21.226+0000 7f1fdea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:21.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:21Z"} 2024-03-04T17:14:21.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:21Z"} 2024-03-04T17:14:21.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:21.246 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:21.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:21.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:14:26.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:26.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:26.249 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:26.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:26.363+0000 7fd63028c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:26.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:26.364+0000 7fd63028c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:26.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:26Z"} 2024-03-04T17:14:26.386 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:26Z"} 2024-03-04T17:14:26.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:26.386 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:26.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:26.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.088500_457.1 2024-03-04T17:14:31.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:31.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:31.388 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:31.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:31.437 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:31.514+0000 7f2be4c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:31.515+0000 7f2be4c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:31.540 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:31Z"} 2024-03-04T17:14:31.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:31Z"} 2024-03-04T17:14:31.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:31.541 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:31.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:31.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T17:14:36.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:36.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:36.543 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:36.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:36.592 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:36.654+0000 7f5843e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:36.655+0000 7f5843e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:36.671 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:36Z"} 2024-03-04T17:14:36.672 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:36Z"} 2024-03-04T17:14:36.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:36.672 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:36.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:36.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T17:14:41.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:41.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:41.673 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:41.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:41.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:41.791 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:41.797+0000 7fa9dff29800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:41.792 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:41.798+0000 7fa9dff29800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:41.818 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:41Z"} 2024-03-04T17:14:41.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:41Z"} 2024-03-04T17:14:41.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:41.819 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:41.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:41.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.860903_435.1 2024-03-04T17:14:46.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:46.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:46.821 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:46.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:46.952+0000 7f8a5a01e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:46.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:46.953+0000 7f8a5a01e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:46.985 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:46Z"} 2024-03-04T17:14:46.985 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:46Z"} 2024-03-04T17:14:46.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:46.986 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:46.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:46.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:14:51.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:51.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:51.988 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:52.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:52.101+0000 7f72b4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:52.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:52.102+0000 7f72b4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:52.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:52Z"} 2024-03-04T17:14:52.125 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:52Z"} 2024-03-04T17:14:52.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:52.125 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:52.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:52.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.951904_442.1 2024-03-04T17:14:57.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:57.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:14:57.127 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:14:57.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:14:57.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:14:57.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:57.243+0000 7fa5c4d20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:57.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:14:57.244+0000 7fa5c4d20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:14:57.265 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:57Z"} 2024-03-04T17:14:57.265 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:57Z"} 2024-03-04T17:14:57.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:14:57.266 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:57.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:14:57.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:15:02.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:02.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:02.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:02.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:02.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:02.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:02.371+0000 7f1abda38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:02.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:02.372+0000 7f1abda38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:02.387 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:02Z"} 2024-03-04T17:15:02.388 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:02Z"} 2024-03-04T17:15:02.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:02.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:02.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:02.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T17:15:07.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:07.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:07.390 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:07.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:07.505+0000 7f0f6402e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:07.505+0000 7f0f6402e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:07.523 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:07Z"} 2024-03-04T17:15:07.523 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:07Z"} 2024-03-04T17:15:07.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:07.523 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:07.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:07.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.948368_440.1 2024-03-04T17:15:12.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:12.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:12.525 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:12.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:12.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:12.654 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:12.660+0000 7f8be1d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:12.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:12.661+0000 7f8be1d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:12.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:12Z"} 2024-03-04T17:15:12.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:12Z"} 2024-03-04T17:15:12.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:12.682 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:12.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:12.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:15:17.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:17.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:17.685 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:17.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:17.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:17.795 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:17.801+0000 7f5f04d1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:17.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:17.802+0000 7f5f04d1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:17.822 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:17Z"} 2024-03-04T17:15:17.822 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:17Z"} 2024-03-04T17:15:17.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:17.822 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:17.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:17.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570370.013079_452.1 2024-03-04T17:15:22.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:22.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:22.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:22.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:22.876 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:22.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:22.944+0000 7f5028931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:22.940 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:22.945+0000 7f5028931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:22Z"} 2024-03-04T17:15:22.966 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:22Z"} 2024-03-04T17:15:22.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:22.966 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:22.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:22.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:15:27.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:27.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:27.971 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:28.018 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:28.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:28.083 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:28.088+0000 7f4c75333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:28.083 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:28.089+0000 7f4c75333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:28.109 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:28Z"} 2024-03-04T17:15:28.109 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:28Z"} 2024-03-04T17:15:28.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:28.109 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:28.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:28.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:15:33.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:33.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:33.111 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:33.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:33.233+0000 7f8ba2137800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:33.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:33.234+0000 7f8ba2137800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:33.258 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:33Z"} 2024-03-04T17:15:33.258 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:33Z"} 2024-03-04T17:15:33.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:33.259 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:33.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:33.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:15:38.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:38.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:38.260 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:38.310 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:38.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:38.387+0000 7fa1a4731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:38.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:38.387+0000 7fa1a4731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:38.412 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:38Z"} 2024-03-04T17:15:38.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:38Z"} 2024-03-04T17:15:38.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:38.412 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:38.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:38.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.994542_448.1 2024-03-04T17:15:43.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:43.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:43.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:43.535+0000 7f043fc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:43.536+0000 7f043fc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:43.553 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:43Z"} 2024-03-04T17:15:43.553 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:43Z"} 2024-03-04T17:15:43.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.218063_467.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.218063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:43.554 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:43.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.218063_467.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:43.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.218063_467.1 2024-03-04T17:15:48.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:48.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:48.556 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:48.607 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:48.607 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:48.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:48.663+0000 7f3d6ebaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:48.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:48.664+0000 7f3d6ebaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:48.682 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:48Z"} 2024-03-04T17:15:48.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:48Z"} 2024-03-04T17:15:48.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:48.683 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:48.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:48.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.155110_458.1 2024-03-04T17:15:53.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:53.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:53.685 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:53.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:53.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:53.794 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:53.799+0000 7f103ea3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:53.795 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:53.800+0000 7f103ea3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:53.827 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:53Z"} 2024-03-04T17:15:53.828 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:53Z"} 2024-03-04T17:15:53.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:53.828 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:53.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:53.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T17:15:58.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:58.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:15:58.830 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:15:58.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:15:58.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:15:58.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:58.961+0000 7fdf34e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:58.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:15:58.962+0000 7fdf34e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:15:58.982 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:58Z"} 2024-03-04T17:15:58.982 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:58Z"} 2024-03-04T17:15:58.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:15:58.982 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:58.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:15:58.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T17:16:03.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:03.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:03.984 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:04.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:04.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:04.113 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:04.118+0000 7f4d7de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:04.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:04.119+0000 7f4d7de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:04.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:04Z"} 2024-03-04T17:16:04.146 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:04Z"} 2024-03-04T17:16:04.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:04.147 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:04.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:04.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T17:16:09.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:09.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:09.149 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:09.201 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:09.201 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:09.270+0000 7fae09816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:09.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:09.271+0000 7fae09816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:09.293 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:09Z"} 2024-03-04T17:16:09.293 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:09Z"} 2024-03-04T17:16:09.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:09.293 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:09.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:09.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.934048_439.1 2024-03-04T17:16:14.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:14.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:14.295 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:14.344 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:14.344 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:14.410 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:14.415+0000 7f9e20d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:14.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:14.416+0000 7f9e20d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:14Z"} 2024-03-04T17:16:14.437 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:14Z"} 2024-03-04T17:16:14.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:14.437 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:14.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:14.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.189913_462.1 2024-03-04T17:16:19.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:19.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:19.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:19.487 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:19.487 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:19.549 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:19.554+0000 7fdccc630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:19.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:19.555+0000 7fdccc630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:19.581 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:19Z"} 2024-03-04T17:16:19.581 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:19Z"} 2024-03-04T17:16:19.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:19.581 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:19.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:19.582 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.948368_440.1 2024-03-04T17:16:24.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:24.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:24.583 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:24.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:24.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:24.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:24.688+0000 7fa7a2815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:24.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:24.689+0000 7fa7a2815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:24.708 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:24Z"} 2024-03-04T17:16:24.709 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:24Z"} 2024-03-04T17:16:24.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:24.709 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:24.709 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:24.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T17:16:29.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:29.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:29.711 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:29.924 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:29.929+0000 7f872549d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:29.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:29.934+0000 7f872549d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:29.982 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:29Z"} 2024-03-04T17:16:29.982 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:29Z"} 2024-03-04T17:16:29.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:29.983 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:29.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:29.983 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T17:16:34.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:34.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:34.985 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:35.056 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:35.056 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:35.123+0000 7f6397431800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:35.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:35.124+0000 7f6397431800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:35.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:35Z"} 2024-03-04T17:16:35.147 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:35Z"} 2024-03-04T17:16:35.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:35.147 DEBUG:rgw_multi.tests:log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:35.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:35.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T17:16:40.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-03-04T17:16:40.153 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-61 2024-03-04T17:16:40.153 DEBUG:boto:path=/sbnvju-61/ 2024-03-04T17:16:40.153 DEBUG:boto:auth_path=/sbnvju-61/ 2024-03-04T17:16:40.153 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.153 DEBUG:boto:Path: /sbnvju-61/ 2024-03-04T17:16:40.153 DEBUG:boto:Data: 2024-03-04T17:16:40.154 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.154 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.154 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.154 DEBUG:boto:Params: {} 2024-03-04T17:16:40.154 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:16:40.155 DEBUG:boto:Token: None 2024-03-04T17:16:40.155 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/ 2024-03-04T17:16:40.155 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:hhLCTjC1f1fDJ+oQyyA+u/A2C8g= 2024-03-04T17:16:40.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hhLCTjC1f1fDJ+oQyyA+u/A2C8g=', 'Content-Length': '0'} 2024-03-04T17:16:40.174 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f18900e06d10ec48-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.174 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-62 2024-03-04T17:16:40.175 DEBUG:boto:path=/sbnvju-62/ 2024-03-04T17:16:40.175 DEBUG:boto:auth_path=/sbnvju-62/ 2024-03-04T17:16:40.175 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.175 DEBUG:boto:Path: /sbnvju-62/ 2024-03-04T17:16:40.175 DEBUG:boto:Data: 2024-03-04T17:16:40.175 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.175 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.175 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.175 DEBUG:boto:Params: {} 2024-03-04T17:16:40.175 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:16:40.176 DEBUG:boto:Token: None 2024-03-04T17:16:40.176 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/ 2024-03-04T17:16:40.176 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ijyKp7E2cgZL02krlImGwo6kTq8= 2024-03-04T17:16:40.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ijyKp7E2cgZL02krlImGwo6kTq8=', 'Content-Length': '0'} 2024-03-04T17:16:40.213 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f53de1578e3947f-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.213 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-63 2024-03-04T17:16:40.214 DEBUG:boto:path=/sbnvju-63/ 2024-03-04T17:16:40.214 DEBUG:boto:auth_path=/sbnvju-63/ 2024-03-04T17:16:40.214 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.214 DEBUG:boto:Path: /sbnvju-63/ 2024-03-04T17:16:40.214 DEBUG:boto:Data: 2024-03-04T17:16:40.214 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.214 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.214 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.214 DEBUG:boto:Params: {} 2024-03-04T17:16:40.214 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:16:40.215 DEBUG:boto:Token: None 2024-03-04T17:16:40.215 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/ 2024-03-04T17:16:40.215 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:mnKjxPMU/nv5l3gF3EHNRddYBkk= 2024-03-04T17:16:40.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mnKjxPMU/nv5l3gF3EHNRddYBkk=', 'Content-Length': '0'} 2024-03-04T17:16:40.243 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ab8ed325ad5c528-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.244 DEBUG:boto:path=/sbnvju-61/ 2024-03-04T17:16:40.244 DEBUG:boto:auth_path=/sbnvju-61/ 2024-03-04T17:16:40.244 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.244 DEBUG:boto:Path: /sbnvju-61/ 2024-03-04T17:16:40.244 DEBUG:boto:Data: 2024-03-04T17:16:40.244 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.244 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.244 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.245 DEBUG:boto:Params: {} 2024-03-04T17:16:40.245 DEBUG:boto:Token: None 2024-03-04T17:16:40.245 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/ 2024-03-04T17:16:40.245 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T17:16:40.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T17:16:40.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002357bbda9c7dc435-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.248 DEBUG:boto:path=/sbnvju-61/obj1 2024-03-04T17:16:40.248 DEBUG:boto:auth_path=/sbnvju-61/obj1 2024-03-04T17:16:40.248 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.248 DEBUG:boto:Path: /sbnvju-61/obj1 2024-03-04T17:16:40.248 DEBUG:boto:Data: 2024-03-04T17:16:40.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.248 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.249 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.249 DEBUG:boto:Params: {} 2024-03-04T17:16:40.249 DEBUG:boto:Token: None 2024-03-04T17:16:40.249 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/obj1 2024-03-04T17:16:40.249 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kJzY2m2p4eki0fsHvNR5MQptQEU= 2024-03-04T17:16:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kJzY2m2p4eki0fsHvNR5MQptQEU='} 2024-03-04T17:16:40.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3629b655009a7a0-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.256 DEBUG:boto:path=/sbnvju-61/ 2024-03-04T17:16:40.256 DEBUG:boto:auth_path=/sbnvju-61/ 2024-03-04T17:16:40.256 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.256 DEBUG:boto:Path: /sbnvju-61/ 2024-03-04T17:16:40.256 DEBUG:boto:Data: 2024-03-04T17:16:40.256 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.256 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.256 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.256 DEBUG:boto:Params: {} 2024-03-04T17:16:40.256 DEBUG:boto:Token: None 2024-03-04T17:16:40.257 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/ 2024-03-04T17:16:40.257 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T17:16:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T17:16:40.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af5db6cba5ba8d1-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.259 DEBUG:boto:path=/sbnvju-61/obj2 2024-03-04T17:16:40.259 DEBUG:boto:auth_path=/sbnvju-61/obj2 2024-03-04T17:16:40.259 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.260 DEBUG:boto:Path: /sbnvju-61/obj2 2024-03-04T17:16:40.260 DEBUG:boto:Data: 2024-03-04T17:16:40.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.260 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.260 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.260 DEBUG:boto:Params: {} 2024-03-04T17:16:40.260 DEBUG:boto:Token: None 2024-03-04T17:16:40.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/obj2 2024-03-04T17:16:40.260 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:4U59cYN9pCmfYQ/obuOecZXSHsM= 2024-03-04T17:16:40.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4U59cYN9pCmfYQ/obuOecZXSHsM='} 2024-03-04T17:16:40.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cb7891d58386c31-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.266 DEBUG:boto:path=/sbnvju-61/ 2024-03-04T17:16:40.266 DEBUG:boto:auth_path=/sbnvju-61/ 2024-03-04T17:16:40.266 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.267 DEBUG:boto:Path: /sbnvju-61/ 2024-03-04T17:16:40.267 DEBUG:boto:Data: 2024-03-04T17:16:40.267 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.267 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.267 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.267 DEBUG:boto:Params: {} 2024-03-04T17:16:40.267 DEBUG:boto:Token: None 2024-03-04T17:16:40.267 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/ 2024-03-04T17:16:40.267 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T17:16:40.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T17:16:40.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ccb959cc06a5ea2-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.270 DEBUG:boto:path=/sbnvju-61/obj3 2024-03-04T17:16:40.270 DEBUG:boto:auth_path=/sbnvju-61/obj3 2024-03-04T17:16:40.270 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.270 DEBUG:boto:Path: /sbnvju-61/obj3 2024-03-04T17:16:40.270 DEBUG:boto:Data: 2024-03-04T17:16:40.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.270 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.270 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.270 DEBUG:boto:Params: {} 2024-03-04T17:16:40.270 DEBUG:boto:Token: None 2024-03-04T17:16:40.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/obj3 2024-03-04T17:16:40.271 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5585jV42lQDJzf1fXAjE9urUYhY= 2024-03-04T17:16:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5585jV42lQDJzf1fXAjE9urUYhY='} 2024-03-04T17:16:40.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1a3129babacad07-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.276 DEBUG:boto:path=/sbnvju-61/ 2024-03-04T17:16:40.276 DEBUG:boto:auth_path=/sbnvju-61/ 2024-03-04T17:16:40.276 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.276 DEBUG:boto:Path: /sbnvju-61/ 2024-03-04T17:16:40.276 DEBUG:boto:Data: 2024-03-04T17:16:40.276 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.277 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.277 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.277 DEBUG:boto:Params: {} 2024-03-04T17:16:40.277 DEBUG:boto:Token: None 2024-03-04T17:16:40.277 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/ 2024-03-04T17:16:40.277 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T17:16:40.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T17:16:40.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a91a1c793e32d40-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.279 DEBUG:boto:path=/sbnvju-61/obj4 2024-03-04T17:16:40.279 DEBUG:boto:auth_path=/sbnvju-61/obj4 2024-03-04T17:16:40.280 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.280 DEBUG:boto:Path: /sbnvju-61/obj4 2024-03-04T17:16:40.280 DEBUG:boto:Data: 2024-03-04T17:16:40.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:16:40.280 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.280 DEBUG:boto:Params: {} 2024-03-04T17:16:40.280 DEBUG:boto:Token: None 2024-03-04T17:16:40.280 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-61/obj4 2024-03-04T17:16:40.280 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:qPSEBSMHHlskUq+S4htfDRajIyM= 2024-03-04T17:16:40.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qPSEBSMHHlskUq+S4htfDRajIyM='} 2024-03-04T17:16:40.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039250795f796b145-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.324 DEBUG:boto:path=/sbnvju-62/ 2024-03-04T17:16:40.325 DEBUG:boto:auth_path=/sbnvju-62/ 2024-03-04T17:16:40.325 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.325 DEBUG:boto:Path: /sbnvju-62/ 2024-03-04T17:16:40.325 DEBUG:boto:Data: 2024-03-04T17:16:40.325 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.325 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.325 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.325 DEBUG:boto:Params: {} 2024-03-04T17:16:40.325 DEBUG:boto:Token: None 2024-03-04T17:16:40.325 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/ 2024-03-04T17:16:40.326 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T17:16:40.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T17:16:40.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b51998707529d2-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.328 DEBUG:boto:path=/sbnvju-62/obj1 2024-03-04T17:16:40.328 DEBUG:boto:auth_path=/sbnvju-62/obj1 2024-03-04T17:16:40.328 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.328 DEBUG:boto:Path: /sbnvju-62/obj1 2024-03-04T17:16:40.328 DEBUG:boto:Data: 2024-03-04T17:16:40.328 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.329 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.329 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.329 DEBUG:boto:Params: {} 2024-03-04T17:16:40.329 DEBUG:boto:Token: None 2024-03-04T17:16:40.329 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/obj1 2024-03-04T17:16:40.329 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:6oCTTk/eoIQlIZkGVdsEMz5KeA8= 2024-03-04T17:16:40.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oCTTk/eoIQlIZkGVdsEMz5KeA8='} 2024-03-04T17:16:40.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000332b0016cb7a836e-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.341 DEBUG:boto:path=/sbnvju-62/ 2024-03-04T17:16:40.341 DEBUG:boto:auth_path=/sbnvju-62/ 2024-03-04T17:16:40.341 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.341 DEBUG:boto:Path: /sbnvju-62/ 2024-03-04T17:16:40.341 DEBUG:boto:Data: 2024-03-04T17:16:40.341 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.342 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.342 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.342 DEBUG:boto:Params: {} 2024-03-04T17:16:40.342 DEBUG:boto:Token: None 2024-03-04T17:16:40.342 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/ 2024-03-04T17:16:40.342 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T17:16:40.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T17:16:40.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c3300d8002e8e13-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.344 DEBUG:boto:path=/sbnvju-62/obj2 2024-03-04T17:16:40.344 DEBUG:boto:auth_path=/sbnvju-62/obj2 2024-03-04T17:16:40.345 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.345 DEBUG:boto:Path: /sbnvju-62/obj2 2024-03-04T17:16:40.345 DEBUG:boto:Data: 2024-03-04T17:16:40.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.345 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.345 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.345 DEBUG:boto:Params: {} 2024-03-04T17:16:40.345 DEBUG:boto:Token: None 2024-03-04T17:16:40.345 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/obj2 2024-03-04T17:16:40.345 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:EzD7ujZ5Dk9Wb1iNj+02k/xP61c= 2024-03-04T17:16:40.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EzD7ujZ5Dk9Wb1iNj+02k/xP61c='} 2024-03-04T17:16:40.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e09a082fbee0dc-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.372 DEBUG:boto:path=/sbnvju-62/ 2024-03-04T17:16:40.372 DEBUG:boto:auth_path=/sbnvju-62/ 2024-03-04T17:16:40.372 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.372 DEBUG:boto:Path: /sbnvju-62/ 2024-03-04T17:16:40.372 DEBUG:boto:Data: 2024-03-04T17:16:40.372 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.372 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.372 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.372 DEBUG:boto:Params: {} 2024-03-04T17:16:40.373 DEBUG:boto:Token: None 2024-03-04T17:16:40.373 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/ 2024-03-04T17:16:40.373 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T17:16:40.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T17:16:40.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dd97678887da140-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.381 DEBUG:boto:path=/sbnvju-62/obj3 2024-03-04T17:16:40.381 DEBUG:boto:auth_path=/sbnvju-62/obj3 2024-03-04T17:16:40.381 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.381 DEBUG:boto:Path: /sbnvju-62/obj3 2024-03-04T17:16:40.381 DEBUG:boto:Data: 2024-03-04T17:16:40.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.381 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.381 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.382 DEBUG:boto:Params: {} 2024-03-04T17:16:40.382 DEBUG:boto:Token: None 2024-03-04T17:16:40.382 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/obj3 2024-03-04T17:16:40.382 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:nCprvWwXht/gV2RHA2N798FWbO8= 2024-03-04T17:16:40.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nCprvWwXht/gV2RHA2N798FWbO8='} 2024-03-04T17:16:40.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c14309132b08af37-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.424 DEBUG:boto:path=/sbnvju-62/ 2024-03-04T17:16:40.424 DEBUG:boto:auth_path=/sbnvju-62/ 2024-03-04T17:16:40.424 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.425 DEBUG:boto:Path: /sbnvju-62/ 2024-03-04T17:16:40.425 DEBUG:boto:Data: 2024-03-04T17:16:40.425 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.425 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.425 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.425 DEBUG:boto:Params: {} 2024-03-04T17:16:40.425 DEBUG:boto:Token: None 2024-03-04T17:16:40.425 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/ 2024-03-04T17:16:40.425 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T17:16:40.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T17:16:40.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7ef0f5b5ff9c157-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.430 DEBUG:boto:path=/sbnvju-62/obj4 2024-03-04T17:16:40.430 DEBUG:boto:auth_path=/sbnvju-62/obj4 2024-03-04T17:16:40.430 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.430 DEBUG:boto:Path: /sbnvju-62/obj4 2024-03-04T17:16:40.430 DEBUG:boto:Data: 2024-03-04T17:16:40.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.430 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:16:40.430 DEBUG:boto:Port: 8000 2024-03-04T17:16:40.430 DEBUG:boto:Params: {} 2024-03-04T17:16:40.430 DEBUG:boto:Token: None 2024-03-04T17:16:40.431 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-62/obj4 2024-03-04T17:16:40.431 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CloIwqoLxajq1ax6oBnWTrTlHdg= 2024-03-04T17:16:40.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CloIwqoLxajq1ax6oBnWTrTlHdg='} 2024-03-04T17:16:40.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061989dbd4ed6fbfb-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.452 DEBUG:boto:path=/sbnvju-63/ 2024-03-04T17:16:40.452 DEBUG:boto:auth_path=/sbnvju-63/ 2024-03-04T17:16:40.452 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.452 DEBUG:boto:Path: /sbnvju-63/ 2024-03-04T17:16:40.452 DEBUG:boto:Data: 2024-03-04T17:16:40.453 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.453 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.453 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.453 DEBUG:boto:Params: {} 2024-03-04T17:16:40.453 DEBUG:boto:Token: None 2024-03-04T17:16:40.453 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/ 2024-03-04T17:16:40.453 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T17:16:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T17:16:40.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044ca7a0efefe427e-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.456 DEBUG:boto:path=/sbnvju-63/obj1 2024-03-04T17:16:40.456 DEBUG:boto:auth_path=/sbnvju-63/obj1 2024-03-04T17:16:40.456 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.456 DEBUG:boto:Path: /sbnvju-63/obj1 2024-03-04T17:16:40.456 DEBUG:boto:Data: 2024-03-04T17:16:40.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.456 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.456 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.456 DEBUG:boto:Params: {} 2024-03-04T17:16:40.457 DEBUG:boto:Token: None 2024-03-04T17:16:40.457 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/obj1 2024-03-04T17:16:40.457 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Y/oOq02/GGm8zsKqyxy5LHQ9R58= 2024-03-04T17:16:40.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y/oOq02/GGm8zsKqyxy5LHQ9R58='} 2024-03-04T17:16:40.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c9bd1a0615215aa-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.463 DEBUG:boto:path=/sbnvju-63/ 2024-03-04T17:16:40.463 DEBUG:boto:auth_path=/sbnvju-63/ 2024-03-04T17:16:40.463 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.463 DEBUG:boto:Path: /sbnvju-63/ 2024-03-04T17:16:40.464 DEBUG:boto:Data: 2024-03-04T17:16:40.464 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.464 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.464 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.464 DEBUG:boto:Params: {} 2024-03-04T17:16:40.464 DEBUG:boto:Token: None 2024-03-04T17:16:40.464 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/ 2024-03-04T17:16:40.464 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T17:16:40.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T17:16:40.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3fd20728ec9c500-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.467 DEBUG:boto:path=/sbnvju-63/obj2 2024-03-04T17:16:40.467 DEBUG:boto:auth_path=/sbnvju-63/obj2 2024-03-04T17:16:40.467 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.467 DEBUG:boto:Path: /sbnvju-63/obj2 2024-03-04T17:16:40.467 DEBUG:boto:Data: 2024-03-04T17:16:40.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.467 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.467 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.468 DEBUG:boto:Params: {} 2024-03-04T17:16:40.468 DEBUG:boto:Token: None 2024-03-04T17:16:40.468 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/obj2 2024-03-04T17:16:40.468 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:iSVlsue2voyFYFrBy4VtMgALB4Q= 2024-03-04T17:16:40.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSVlsue2voyFYFrBy4VtMgALB4Q='} 2024-03-04T17:16:40.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfbee6d0dd4f2168-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.473 DEBUG:boto:path=/sbnvju-63/ 2024-03-04T17:16:40.473 DEBUG:boto:auth_path=/sbnvju-63/ 2024-03-04T17:16:40.474 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.474 DEBUG:boto:Path: /sbnvju-63/ 2024-03-04T17:16:40.474 DEBUG:boto:Data: 2024-03-04T17:16:40.474 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.474 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.474 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.474 DEBUG:boto:Params: {} 2024-03-04T17:16:40.474 DEBUG:boto:Token: None 2024-03-04T17:16:40.474 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/ 2024-03-04T17:16:40.474 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T17:16:40.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T17:16:40.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbb4f4ccdb977e94-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.477 DEBUG:boto:path=/sbnvju-63/obj3 2024-03-04T17:16:40.477 DEBUG:boto:auth_path=/sbnvju-63/obj3 2024-03-04T17:16:40.477 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.477 DEBUG:boto:Path: /sbnvju-63/obj3 2024-03-04T17:16:40.477 DEBUG:boto:Data: 2024-03-04T17:16:40.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.478 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.478 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.478 DEBUG:boto:Params: {} 2024-03-04T17:16:40.478 DEBUG:boto:Token: None 2024-03-04T17:16:40.478 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/obj3 2024-03-04T17:16:40.478 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:OP3i0fxF0SoHXTaBQH+tJq5pJS0= 2024-03-04T17:16:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OP3i0fxF0SoHXTaBQH+tJq5pJS0='} 2024-03-04T17:16:40.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094f664a2a4baa2cf-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.520 DEBUG:boto:path=/sbnvju-63/ 2024-03-04T17:16:40.520 DEBUG:boto:auth_path=/sbnvju-63/ 2024-03-04T17:16:40.520 DEBUG:boto:Method: HEAD 2024-03-04T17:16:40.521 DEBUG:boto:Path: /sbnvju-63/ 2024-03-04T17:16:40.521 DEBUG:boto:Data: 2024-03-04T17:16:40.521 DEBUG:boto:Headers: {} 2024-03-04T17:16:40.521 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.521 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.521 DEBUG:boto:Params: {} 2024-03-04T17:16:40.521 DEBUG:boto:Token: None 2024-03-04T17:16:40.521 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/ 2024-03-04T17:16:40.521 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T17:16:40.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T17:16:40.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e52f86c991ba534-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.524 DEBUG:boto:path=/sbnvju-63/obj4 2024-03-04T17:16:40.524 DEBUG:boto:auth_path=/sbnvju-63/obj4 2024-03-04T17:16:40.524 DEBUG:boto:Method: PUT 2024-03-04T17:16:40.524 DEBUG:boto:Path: /sbnvju-63/obj4 2024-03-04T17:16:40.525 DEBUG:boto:Data: 2024-03-04T17:16:40.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T17:16:40.525 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:16:40.525 DEBUG:boto:Port: 8001 2024-03-04T17:16:40.525 DEBUG:boto:Params: {} 2024-03-04T17:16:40.525 DEBUG:boto:Token: None 2024-03-04T17:16:40.525 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 04 Mar 2024 17:16:40 GMT /sbnvju-63/obj4 2024-03-04T17:16:40.525 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:oKh9wQ0r0F8KTD+ANl8yR3q2DM8= 2024-03-04T17:16:40.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oKh9wQ0r0F8KTD+ANl8yR3q2DM8='} 2024-03-04T17:16:40.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0f914daddaf7f25-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:16:40.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:40.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:40.531 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:40.577 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:40.577 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:40.623 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:16:40.643 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709572600.239122_819.1","last_update":"2024-03-04T17:16:40.239122Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:16:40.643 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:40.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:16:40.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:40.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:40.644 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:40.689 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:40.689 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:40.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:40.752+0000 7f4837c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:40.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:40.753+0000 7f4837c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:40.774 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:40Z"} 2024-03-04T17:16:40.774 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:40Z"} 2024-03-04T17:16:40.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:40.775 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:40.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:40.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T17:16:45.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:45.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:45.776 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:45.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:45.884+0000 7fec0b816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:45.885+0000 7fec0b816800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:45.904 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:45Z"} 2024-03-04T17:16:45.905 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:45Z"} 2024-03-04T17:16:45.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:45.905 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:45.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:45.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T17:16:50.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:50.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:50.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:50.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:50.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:51.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:51.029+0000 7f35ddb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:51.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:51.030+0000 7f35ddb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:51.050 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:51Z"} 2024-03-04T17:16:51.050 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:51Z"} 2024-03-04T17:16:51.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:51.050 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:51.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:51.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:16:56.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:56.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:16:56.052 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:16:56.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:16:56.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:16:56.162 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:56.167+0000 7f7fce813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:56.163 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:16:56.168+0000 7f7fce813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:16:56.186 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:56Z"} 2024-03-04T17:16:56.186 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:56Z"} 2024-03-04T17:16:56.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:16:56.186 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:56.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:16:56.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T17:17:01.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:01.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:01.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:01.234 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:01.234 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:01.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:01.300+0000 7f4cd420f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:01.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:01.301+0000 7f4cd420f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:01.319 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:01Z"} 2024-03-04T17:17:01.319 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:01Z"} 2024-03-04T17:17:01.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:01.320 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:01.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:01.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T17:17:06.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:06.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:06.321 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:06.548 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:06.553+0000 7f43c2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:06.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:06.554+0000 7f43c2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:06.579 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:06Z"} 2024-03-04T17:17:06.579 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:06Z"} 2024-03-04T17:17:06.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:06.579 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:06.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:06.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:17:11.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:11.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:11.581 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:11.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:11.627 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:11.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:11.680+0000 7fbf1bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:11.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:11.681+0000 7fbf1bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:11.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:11Z"} 2024-03-04T17:17:11.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:11Z"} 2024-03-04T17:17:11.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:11.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:11.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:11.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.858235_434.1 2024-03-04T17:17:16.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:16.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:16.706 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:16.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:16.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:16.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:16.819+0000 7f644472e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:16.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:16.820+0000 7f644472e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:16.842 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:16Z"} 2024-03-04T17:17:16.843 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:16Z"} 2024-03-04T17:17:16.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:16.843 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:16.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:16.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T17:17:21.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:21.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:21.845 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:21.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:21.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:21.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:21.961+0000 7f00ab630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:21.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:21.961+0000 7f00ab630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:21.987 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:21Z"} 2024-03-04T17:17:21.987 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:21Z"} 2024-03-04T17:17:21.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:21.988 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:21.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:21.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.996984_449.1 2024-03-04T17:17:26.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:26.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:26.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:27.036 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:27.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:27.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:27.096+0000 7f3ca8b35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:27.096+0000 7f3ca8b35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:27.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:27Z"} 2024-03-04T17:17:27.114 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:27Z"} 2024-03-04T17:17:27.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:27.114 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:27.114 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:27.114 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T17:17:32.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:32.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:32.116 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:32.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:32.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:32.234 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:32.238+0000 7f8b42931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:32.234 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:32.239+0000 7f8b42931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:32.258 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:32Z"} 2024-03-04T17:17:32.258 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:32Z"} 2024-03-04T17:17:32.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:32.258 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:32.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:32.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:17:37.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:37.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:37.260 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:37.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:37.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:37.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:37.379+0000 7fdd7c53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:37.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:37.380+0000 7fdd7c53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:37.399 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:37Z"} 2024-03-04T17:17:37.399 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:37Z"} 2024-03-04T17:17:37.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:37.399 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:37.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:37.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T17:17:42.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:42.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:42.401 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:42.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:42.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:42.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:42.520+0000 7fa9d0613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:42.517 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:42.521+0000 7fa9d0613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:42.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:42Z"} 2024-03-04T17:17:42.549 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:42Z"} 2024-03-04T17:17:42.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:42.549 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:42.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:42.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:17:47.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:47.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:47.550 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:47.600 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:47.600 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:47.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:47.668+0000 7fc39214b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:47.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:47.668+0000 7fc39214b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:47.693 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:47Z"} 2024-03-04T17:17:47.693 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:47Z"} 2024-03-04T17:17:47.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:47.694 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:47.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:47.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:17:52.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:52.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:52.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:52.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:52.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:52.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:52.815+0000 7f76f274b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:52.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:52.815+0000 7f76f274b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:52.835 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:52Z"} 2024-03-04T17:17:52.836 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:52Z"} 2024-03-04T17:17:52.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:52.836 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:52.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:52.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T17:17:57.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:57.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:17:57.838 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:17:57.886 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:17:57.886 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:17:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:57.942+0000 7f2a5a127800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:17:57.943+0000 7f2a5a127800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:17:57.969 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:57Z"} 2024-03-04T17:17:57.969 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:57Z"} 2024-03-04T17:17:57.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:17:57.970 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:57.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:17:57.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T17:18:02.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:02.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:02.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:03.022 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:03.022 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:03.077+0000 7f2fa1333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:03.078+0000 7f2fa1333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:03.096 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:03Z"} 2024-03-04T17:18:03.097 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:03Z"} 2024-03-04T17:18:03.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:03.097 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:03.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:03.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:18:08.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:08.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:08.099 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:08.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:08.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:08.229+0000 7f84b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:08.229+0000 7f84b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:08.251 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:08Z"} 2024-03-04T17:18:08.252 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:08Z"} 2024-03-04T17:18:08.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:08.252 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:08.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:08.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:18:13.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:13.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:13.254 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:13.302 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:13.302 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:13.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:13.358+0000 7f324742d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:13.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:13.359+0000 7f324742d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:13.385 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:13Z"} 2024-03-04T17:18:13.386 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:13Z"} 2024-03-04T17:18:13.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:13.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:13.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:13.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T17:18:18.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:18.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:18.388 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:18.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:18.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:18.501 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:18.505+0000 7fa046424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:18.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:18.506+0000 7fa046424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:18.526 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:18Z"} 2024-03-04T17:18:18.527 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:18Z"} 2024-03-04T17:18:18.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:18.527 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:18.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:18.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:18:23.423 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:18:23.426+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:18:23.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:23.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:23.529 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:23.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:23.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:23.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:23.663+0000 7f45a2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:23.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:23.663+0000 7f45a2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:23.731 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:23Z"} 2024-03-04T17:18:23.731 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:23Z"} 2024-03-04T17:18:23.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:23.732 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:23.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:23.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T17:18:28.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:28.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:28.734 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:28.785 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:28.785 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:28.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:28.952+0000 7faf6bc39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:28.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:28.956+0000 7faf6bc39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:29.017 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:28Z"} 2024-03-04T17:18:29.017 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:28Z"} 2024-03-04T17:18:29.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:29.018 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:29.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:29.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.954325_443.1 2024-03-04T17:18:34.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:34.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:34.020 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:34.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:34.160+0000 7f282c731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:34.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:34.161+0000 7f282c731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:34.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:34Z"} 2024-03-04T17:18:34.184 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:34Z"} 2024-03-04T17:18:34.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:34.187 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:34.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:34.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T17:18:39.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:39.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:39.189 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:39.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:39.317+0000 7fc4a932e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:39.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:39.318+0000 7fc4a932e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:39.337 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:39Z"} 2024-03-04T17:18:39.337 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:39Z"} 2024-03-04T17:18:39.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:39.338 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:39.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:39.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T17:18:44.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:44.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:44.339 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:44.387 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:44.387 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:44.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:44.450+0000 7f660871e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:44.447 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:44.451+0000 7f660871e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:44.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:44Z"} 2024-03-04T17:18:44.479 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:44Z"} 2024-03-04T17:18:44.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:44.479 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:44.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:44.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.994542_448.1 2024-03-04T17:18:49.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:49.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:49.481 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:49.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:49.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:49.584 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:49.587+0000 7fe24573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:49.584 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:49.588+0000 7fe24573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:49.608 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:49Z"} 2024-03-04T17:18:49.608 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:49Z"} 2024-03-04T17:18:49.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:49.609 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:49.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:49.609 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T17:18:54.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:54.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:54.611 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:54.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:54.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:54.709 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:54.712+0000 7f8cf7e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:54.710 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:54.713+0000 7f8cf7e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:54.733 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:54Z"} 2024-03-04T17:18:54.734 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:54Z"} 2024-03-04T17:18:54.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:54.734 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:54.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:54.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T17:18:59.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:59.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:18:59.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:18:59.786 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:18:59.787 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:18:59.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:59.843+0000 7f15c812f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:59.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:18:59.844+0000 7f15c812f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:18:59.872 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:59Z"} 2024-03-04T17:18:59.873 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:59Z"} 2024-03-04T17:18:59.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:18:59.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:59.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:18:59.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.951904_442.1 2024-03-04T17:19:04.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:04.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:04.875 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:04.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:04.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:04.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:04.975+0000 7f7b552c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:04.976+0000 7f7b552c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:04.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:04Z"} 2024-03-04T17:19:04.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:04Z"} 2024-03-04T17:19:04.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:04.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:04.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:04.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T17:19:09.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:09.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:09.999 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:10.045 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:10.045 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:10.103 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:10.106+0000 7f9691c15800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:10.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:10.107+0000 7f9691c15800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:10.128 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:10Z"} 2024-03-04T17:19:10.128 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:10Z"} 2024-03-04T17:19:10.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:10.129 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:10.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:10.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:19:15.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:15.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:15.131 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:15.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:15.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:15.252 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:15.255+0000 7f9a153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:15.252 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:15.256+0000 7f9a153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:15.276 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:15Z"} 2024-03-04T17:19:15.276 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:15Z"} 2024-03-04T17:19:15.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:15.276 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:15.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:15.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T17:19:20.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:20.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:20.278 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:20.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:20.378+0000 7fb17d10d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:20.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:20.379+0000 7fb17d10d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:20.397 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:20Z"} 2024-03-04T17:19:20.397 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:20Z"} 2024-03-04T17:19:20.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:20.397 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:20.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:20.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T17:19:21.291 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:19:21.294+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:19:25.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:25.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:25.399 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:25.518+0000 7f1e66e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:25.519+0000 7f1e66e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:25.545 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:25Z"} 2024-03-04T17:19:25.545 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:25Z"} 2024-03-04T17:19:25.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:25.546 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:25.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:25.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570370.013079_452.1 2024-03-04T17:19:30.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:30.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:30.548 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:30.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:30.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:30.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:30.662+0000 7f42f3c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:30.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:30.663+0000 7f42f3c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:30.692 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:30Z"} 2024-03-04T17:19:30.693 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:30Z"} 2024-03-04T17:19:30.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:30.693 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:30.693 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:30.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T17:19:35.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:35.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:35.695 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:35.743 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:35.743 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:35.806+0000 7fa25e32d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:35.807+0000 7fa25e32d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:35.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:35Z"} 2024-03-04T17:19:35.834 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:35Z"} 2024-03-04T17:19:35.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:35.834 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:35.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:35.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T17:19:40.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:40.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:40.836 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:40.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:40.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:40.941+0000 7f769c667800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:40.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:40.942+0000 7f769c667800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:40.964 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:40Z"} 2024-03-04T17:19:40.964 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:40Z"} 2024-03-04T17:19:40.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:40.965 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:40.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:40.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T17:19:45.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:45.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:45.966 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:46.016 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:46.016 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:46.080 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:46.083+0000 7f68c813b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:46.081 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:46.084+0000 7f68c813b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:46.111 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:46Z"} 2024-03-04T17:19:46.111 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:46Z"} 2024-03-04T17:19:46.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:46.111 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:46.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:46.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T17:19:51.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:51.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:51.113 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:51.166 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:51.166 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:51.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:51.226+0000 7fec89533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:51.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:51.227+0000 7fec89533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:51.247 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:51Z"} 2024-03-04T17:19:51.248 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:51Z"} 2024-03-04T17:19:51.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:51.248 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:51.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:51.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T17:19:56.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:56.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:19:56.249 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:19:56.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:19:56.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:19:56.367 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:56.370+0000 7fafce89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:56.368 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:19:56.371+0000 7fafce89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:19:56.395 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:56Z"} 2024-03-04T17:19:56.395 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:56Z"} 2024-03-04T17:19:56.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:19:56.395 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:56.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:19:56.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:20:01.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:01.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:01.397 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:01.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:01.450 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:01.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:01.527+0000 7fa79cf2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:01.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:01.528+0000 7fa79cf2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:01.550 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:01Z"} 2024-03-04T17:20:01.551 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:01Z"} 2024-03-04T17:20:01.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:01.551 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:01.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:01.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T17:20:06.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:06.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:06.552 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:06.600 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:06.600 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:06.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:06.672+0000 7f4dda72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:06.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:06.673+0000 7f4dda72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:06.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:06Z"} 2024-03-04T17:20:06.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:06Z"} 2024-03-04T17:20:06.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:06.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:06.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:06.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.927093_438.1 2024-03-04T17:20:11.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:11.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:11.705 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:11.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:11.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:11.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:11.828+0000 7fad1249d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:11.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:11.829+0000 7fad1249d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:11.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:11Z"} 2024-03-04T17:20:11.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:11Z"} 2024-03-04T17:20:11.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:11.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:11.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:11.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.975603_447.1 2024-03-04T17:20:16.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:16.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:16.926 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:17.007 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:17.007 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:17.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:17.116+0000 7f7549639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:17.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:17.117+0000 7f7549639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:17.137 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:17Z"} 2024-03-04T17:20:17.137 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:17Z"} 2024-03-04T17:20:17.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:17.138 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:17.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:17.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:20:22.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:22.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:22.139 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:22.214 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:22.214 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:22.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:22.301+0000 7f77b7f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:22.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:22.302+0000 7f77b7f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:22.332 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:22Z"} 2024-03-04T17:20:22.332 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:22Z"} 2024-03-04T17:20:22.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:22.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:22.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:22.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:20:27.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:27.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:27.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:27.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:27.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:27.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:27.458+0000 7eff13044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:27.456 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:27.459+0000 7eff13044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:27.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:27Z"} 2024-03-04T17:20:27.480 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:27Z"} 2024-03-04T17:20:27.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:27.480 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:27.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:27.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T17:20:32.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:32.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:32.482 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:32.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:32.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:32.595 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:32.597+0000 7f7c5d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:32.596 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:32.598+0000 7f7c5d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:32.620 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:32Z"} 2024-03-04T17:20:32.620 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:32Z"} 2024-03-04T17:20:32.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:32.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:32.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:32.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T17:20:37.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:37.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:37.622 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:37.670 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:37.671 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:37.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:37.741+0000 7ff029c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:37.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:37.741+0000 7ff029c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:37.766 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:37Z"} 2024-03-04T17:20:37.766 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:37Z"} 2024-03-04T17:20:37.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:37.766 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:37.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:37.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570370.013079_452.1 2024-03-04T17:20:42.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:42.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:42.768 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:42.814 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:42.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:42.890+0000 7f04b412f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:42.891+0000 7f04b412f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:42.919 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:42Z"} 2024-03-04T17:20:42.920 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:42Z"} 2024-03-04T17:20:42.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:42.920 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:42.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:42.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.927093_438.1 2024-03-04T17:20:47.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:47.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:47.922 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:47.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:47.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:48.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:48.025+0000 7f9f4f7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:48.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:48.026+0000 7f9f4f7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:48.047 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:48Z"} 2024-03-04T17:20:48.047 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:48Z"} 2024-03-04T17:20:48.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:48.048 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:48.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:48.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:20:53.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:53.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:53.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:53.162 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:53.164+0000 7f7c6714b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:53.162 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:53.165+0000 7f7c6714b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:53.186 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:53Z"} 2024-03-04T17:20:53.187 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:53Z"} 2024-03-04T17:20:53.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:53.187 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:53.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:53.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.954325_443.1 2024-03-04T17:20:58.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:58.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:20:58.189 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:20:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:20:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:20:58.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:58.298+0000 7fa671242800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:20:58.299+0000 7fa671242800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:20:58.320 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:58Z"} 2024-03-04T17:20:58.320 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:58Z"} 2024-03-04T17:20:58.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:20:58.320 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:58.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:20:58.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T17:21:03.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:03.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:03.322 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:03.427+0000 7fad1d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:03.428+0000 7fad1d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:03.449 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:03Z"} 2024-03-04T17:21:03.449 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:03Z"} 2024-03-04T17:21:03.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:03.449 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:03.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:03.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570372.084490_456.1 2024-03-04T17:21:08.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:08.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:08.451 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:08.499 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:08.499 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:08.549 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:08.551+0000 7f901ba30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:08.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:08.552+0000 7f901ba30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:08.574 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:08Z"} 2024-03-04T17:21:08.574 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:08Z"} 2024-03-04T17:21:08.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:08.575 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:08.575 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:08.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:21:13.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:13.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:13.576 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:13.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:13.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:13.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:13.679+0000 7fcfec331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:13.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:13.680+0000 7fcfec331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:13.700 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:13Z"} 2024-03-04T17:21:13.701 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:13Z"} 2024-03-04T17:21:13.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:13.701 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:13.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:13.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T17:21:18.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:18.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:18.703 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:18.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:18.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:18.806 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:18.809+0000 7fd5f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:18.807 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:18.810+0000 7fd5f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:18.836 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:18Z"} 2024-03-04T17:21:18.836 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:18Z"} 2024-03-04T17:21:18.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:18.836 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:18.837 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:18.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T17:21:23.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:23.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:23.839 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:23.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:23.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:23.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:23.951+0000 7fe56a48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:23.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:23.952+0000 7fe56a48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:23.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:23Z"} 2024-03-04T17:21:23.981 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:23Z"} 2024-03-04T17:21:23.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.078213_520.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.078213Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:23.982 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:23.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.078213_520.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:23.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570680.078213_520.1 2024-03-04T17:21:28.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:28.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:28.983 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:29.031 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:29.032 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:29.093+0000 7f2dd95af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:29.094+0000 7f2dd95af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:29.120 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:29Z"} 2024-03-04T17:21:29.120 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:29Z"} 2024-03-04T17:21:29.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:29.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:29.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:29.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T17:21:34.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:34.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:34.123 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:34.175 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:34.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:34.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:34.242+0000 7fa28a824800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:34.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:34.243+0000 7fa28a824800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:34.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:34Z"} 2024-03-04T17:21:34.275 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:34Z"} 2024-03-04T17:21:34.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:34.275 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:34.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:34.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T17:21:39.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:39.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:39.277 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:39.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:39.383+0000 7fdb3f016800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:39.384+0000 7fdb3f016800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:39.411 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:39Z"} 2024-03-04T17:21:39.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:39Z"} 2024-03-04T17:21:39.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:39.412 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:39.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:39.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:21:44.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:44.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:44.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:44.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:44.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:44.628 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:44.629+0000 7f980cd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:44.630 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:44.632+0000 7f980cd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:44.674 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:44Z"} 2024-03-04T17:21:44.674 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:44Z"} 2024-03-04T17:21:44.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:44.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:44.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:44.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T17:21:49.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-03-04T17:21:49.680 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-64 2024-03-04T17:21:49.680 DEBUG:boto:path=/sbnvju-64/ 2024-03-04T17:21:49.681 DEBUG:boto:auth_path=/sbnvju-64/ 2024-03-04T17:21:49.681 DEBUG:boto:Method: PUT 2024-03-04T17:21:49.681 DEBUG:boto:Path: /sbnvju-64/ 2024-03-04T17:21:49.681 DEBUG:boto:Data: 2024-03-04T17:21:49.681 DEBUG:boto:Headers: {} 2024-03-04T17:21:49.681 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:49.681 DEBUG:boto:Port: 8000 2024-03-04T17:21:49.682 DEBUG:boto:Params: {} 2024-03-04T17:21:49.682 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:21:49.682 DEBUG:boto:Token: None 2024-03-04T17:21:49.683 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:21:49 GMT /sbnvju-64/ 2024-03-04T17:21:49.683 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:CEYLftg7jED1JErGZIwefD3VUYI= 2024-03-04T17:21:49.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CEYLftg7jED1JErGZIwefD3VUYI=', 'Content-Length': '0'} 2024-03-04T17:21:49.705 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003166221d0afb6959-0065e6032d-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:49.705 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=sbnvju-65 2024-03-04T17:21:49.705 DEBUG:boto:path=/sbnvju-65/ 2024-03-04T17:21:49.705 DEBUG:boto:auth_path=/sbnvju-65/ 2024-03-04T17:21:49.705 DEBUG:boto:Method: PUT 2024-03-04T17:21:49.705 DEBUG:boto:Path: /sbnvju-65/ 2024-03-04T17:21:49.705 DEBUG:boto:Data: 2024-03-04T17:21:49.706 DEBUG:boto:Headers: {} 2024-03-04T17:21:49.706 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T17:21:49.706 DEBUG:boto:Port: 8000 2024-03-04T17:21:49.706 DEBUG:boto:Params: {} 2024-03-04T17:21:49.706 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:21:49.706 DEBUG:boto:Token: None 2024-03-04T17:21:49.706 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:21:49 GMT /sbnvju-65/ 2024-03-04T17:21:49.706 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:IBomHfCt0HBbvYoy0fyRkqJePHs= 2024-03-04T17:21:49.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IBomHfCt0HBbvYoy0fyRkqJePHs=', 'Content-Length': '0'} 2024-03-04T17:21:49.745 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4edc14516a3876e-0065e6032d-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:49.745 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=sbnvju-66 2024-03-04T17:21:49.745 DEBUG:boto:path=/sbnvju-66/ 2024-03-04T17:21:49.745 DEBUG:boto:auth_path=/sbnvju-66/ 2024-03-04T17:21:49.746 DEBUG:boto:Method: PUT 2024-03-04T17:21:49.746 DEBUG:boto:Path: /sbnvju-66/ 2024-03-04T17:21:49.746 DEBUG:boto:Data: 2024-03-04T17:21:49.746 DEBUG:boto:Headers: {} 2024-03-04T17:21:49.746 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T17:21:49.747 DEBUG:boto:Port: 8001 2024-03-04T17:21:49.747 DEBUG:boto:Params: {} 2024-03-04T17:21:49.747 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T17:21:49.748 DEBUG:boto:Token: None 2024-03-04T17:21:49.748 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:21:49 GMT /sbnvju-66/ 2024-03-04T17:21:49.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:ped9zcZDxZWxZyOE8O/9TY/8T7k= 2024-03-04T17:21:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ped9zcZDxZWxZyOE8O/9TY/8T7k=', 'Content-Length': '0'} 2024-03-04T17:21:49.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e32b78d169679a90-0065e6032d-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:49.781 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:49.781 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:49.781 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploads 2024-03-04T17:21:49.782 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploads 2024-03-04T17:21:49.782 DEBUG:boto:Method: POST 2024-03-04T17:21:49.782 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploads 2024-03-04T17:21:49.782 DEBUG:boto:Data: 2024-03-04T17:21:49.782 DEBUG:boto:Headers: {} 2024-03-04T17:21:49.782 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:49.782 DEBUG:boto:Port: 8000 2024-03-04T17:21:49.782 DEBUG:boto:Params: {} 2024-03-04T17:21:49.782 DEBUG:boto:Token: None 2024-03-04T17:21:49.782 DEBUG:boto:StringToSign: POST Mon, 04 Mar 2024 17:21:49 GMT /sbnvju-64/MULTIPART?uploads 2024-03-04T17:21:49.783 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:c6HudOOCO44uKcFMnsJnFWNNeIE= 2024-03-04T17:21:49.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c6HudOOCO44uKcFMnsJnFWNNeIE=', 'Content-Length': '0'} 2024-03-04T17:21:51.210 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb315d133ee963df-0065e6032d-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.210 DEBUG:boto:b'sbnvju-64MULTIPART2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2' 2024-03-04T17:21:51.256 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.257 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.257 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T17:21:51.257 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T17:21:51.257 DEBUG:boto:Method: PUT 2024-03-04T17:21:51.257 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T17:21:51.257 DEBUG:boto:Data: 2024-03-04T17:21:51.257 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T17:21:51.257 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.257 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.257 DEBUG:boto:Params: {} 2024-03-04T17:21:51.258 DEBUG:boto:Token: None 2024-03-04T17:21:51.258 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?partNumber=1&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.258 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:prVxvh/BP37jaEmBaXL1oq/rM5Y= 2024-03-04T17:21:51.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:prVxvh/BP37jaEmBaXL1oq/rM5Y='} 2024-03-04T17:21:51.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d538b32d26173d34-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.351 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.352 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.352 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T17:21:51.352 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T17:21:51.352 DEBUG:boto:Method: PUT 2024-03-04T17:21:51.352 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T17:21:51.352 DEBUG:boto:Data: 2024-03-04T17:21:51.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T17:21:51.352 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.352 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.353 DEBUG:boto:Params: {} 2024-03-04T17:21:51.353 DEBUG:boto:Token: None 2024-03-04T17:21:51.353 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?partNumber=2&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.353 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:jAk4IGBYRSaNeVA+n1rtn8TTn50= 2024-03-04T17:21:51.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jAk4IGBYRSaNeVA+n1rtn8TTn50='} 2024-03-04T17:21:51.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000713836097de8379f-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.429 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.429 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.429 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T17:21:51.429 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T17:21:51.429 DEBUG:boto:Method: PUT 2024-03-04T17:21:51.429 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T17:21:51.430 DEBUG:boto:Data: 2024-03-04T17:21:51.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T17:21:51.430 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.430 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.430 DEBUG:boto:Params: {} 2024-03-04T17:21:51.430 DEBUG:boto:Token: None 2024-03-04T17:21:51.430 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?partNumber=3&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.430 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:pxvwniKCLg8XEJdKOZET1gDg3po= 2024-03-04T17:21:51.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pxvwniKCLg8XEJdKOZET1gDg3po='} 2024-03-04T17:21:51.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ceb167fd0781eaa2-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.510 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.510 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.510 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T17:21:51.510 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T17:21:51.510 DEBUG:boto:Method: PUT 2024-03-04T17:21:51.510 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T17:21:51.510 DEBUG:boto:Data: 2024-03-04T17:21:51.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T17:21:51.511 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.511 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.511 DEBUG:boto:Params: {} 2024-03-04T17:21:51.511 DEBUG:boto:Token: None 2024-03-04T17:21:51.511 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?partNumber=4&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.511 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Uo838De1lPXbeVWYHn2wKs3VrYA= 2024-03-04T17:21:51.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uo838De1lPXbeVWYHn2wKs3VrYA='} 2024-03-04T17:21:51.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c5727c08a89c055-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.566 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.566 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.566 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.566 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.567 DEBUG:boto:Method: GET 2024-03-04T17:21:51.567 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.567 DEBUG:boto:Data: 2024-03-04T17:21:51.567 DEBUG:boto:Headers: {} 2024-03-04T17:21:51.567 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.567 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.567 DEBUG:boto:Params: {} 2024-03-04T17:21:51.567 DEBUG:boto:Token: None 2024-03-04T17:21:51.567 DEBUG:boto:StringToSign: GET Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.568 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Tfz00BaRNCM8UDJUgLEMaHG+tsY= 2024-03-04T17:21:51.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tfz00BaRNCM8UDJUgLEMaHG+tsY=', 'Content-Length': '0'} 2024-03-04T17:21:51.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083439ded0b5e3fb1-0065e6032f-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.616 DEBUG:boto:path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.616 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART 2024-03-04T17:21:51.617 DEBUG:boto:path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.617 DEBUG:boto:auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.617 DEBUG:boto:Method: POST 2024-03-04T17:21:51.617 DEBUG:boto:Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.617 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-03-04T17:21:51.617 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-03-04T17:21:51.617 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:21:51.617 DEBUG:boto:Port: 8000 2024-03-04T17:21:51.617 DEBUG:boto:Params: {} 2024-03-04T17:21:51.617 DEBUG:boto:Token: None 2024-03-04T17:21:51.618 DEBUG:boto:StringToSign: POST text/xml Mon, 04 Mar 2024 17:21:51 GMT /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T17:21:51.618 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:T/Eb3wJ+IjDfHryw8o4t+2z5/9A= 2024-03-04T17:21:51.618 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T/Eb3wJ+IjDfHryw8o4t+2z5/9A=', 'Content-Length': '460'} 2024-03-04T17:21:51.628 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2106474bc286131-0065e6032f-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:21:51.629 DEBUG:boto:http://smithi027.front.sepia.ceph.com:8000/sbnvju-64/MULTIPARTsbnvju-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-03-04T17:21:51.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:51.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:51.629 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:21:51.741 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709572909.772492_857.1","last_update":"2024-03-04T17:21:49.772492Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:21:51.741 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:51.741 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:21:51.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:51.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:51.742 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:51.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:51.789 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:51.837 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:51.839+0000 7fdd48a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:51.838 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:51.840+0000 7fdd48a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:51.860 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:51Z"} 2024-03-04T17:21:51.860 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:51Z"} 2024-03-04T17:21:51.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:51.860 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:51.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:51.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.840554_431.1 2024-03-04T17:21:56.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:56.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:21:56.863 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:21:56.909 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:21:56.909 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:21:56.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:56.968+0000 7f5940911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:56.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:21:56.968+0000 7f5940911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:21:56.996 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:56Z"} 2024-03-04T17:21:56.996 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:56Z"} 2024-03-04T17:21:56.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:21:56.996 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:56.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:21:56.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.951904_442.1 2024-03-04T17:22:01.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:01.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:01.998 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:02.045 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:02.045 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:02.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:02.099+0000 7f6b42412800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:02.098 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:02.100+0000 7f6b42412800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:02.121 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:02Z"} 2024-03-04T17:22:02.121 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:02Z"} 2024-03-04T17:22:02.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:02.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:02.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:02.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T17:22:07.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:07.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:07.123 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:07.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:07.224+0000 7ff4f6a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:07.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:07.225+0000 7ff4f6a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:07.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:07Z"} 2024-03-04T17:22:07.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:07Z"} 2024-03-04T17:22:07.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:07.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:07.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:07.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.843902_433.1 2024-03-04T17:22:12.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:12.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:12.252 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:12.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:12.350+0000 7f2e7e530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:12.348 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:12.351+0000 7f2e7e530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:12.373 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:12Z"} 2024-03-04T17:22:12.373 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:12Z"} 2024-03-04T17:22:12.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:12.373 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:12.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:12.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.840554_431.1 2024-03-04T17:22:17.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:17.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:17.375 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:17.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:17.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:17.580 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:17.582+0000 7ff8fb933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:17.581 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:17.583+0000 7ff8fb933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:17.611 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:17Z"} 2024-03-04T17:22:17.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:17Z"} 2024-03-04T17:22:17.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:17.612 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:17.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:17.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.873456_437.1 2024-03-04T17:22:22.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:22.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:22.613 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:22.674 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:22.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:22.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:22.729+0000 7fbc03630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:22.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:22.730+0000 7fbc03630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:22.752 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:22Z"} 2024-03-04T17:22:22.753 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:22Z"} 2024-03-04T17:22:22.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:22.753 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:22.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:22.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.843902_433.1 2024-03-04T17:22:27.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:27.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:27.755 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:27.802 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:27.802 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:27.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:27.876+0000 7f4995644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:27.875 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:27.877+0000 7f4995644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:27.898 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:27Z"} 2024-03-04T17:22:27.898 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:27Z"} 2024-03-04T17:22:27.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:27.899 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:27.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:27.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T17:22:32.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:32.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:32.901 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:32.947 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:32.948 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:33.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:33.004+0000 7f632b03a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:33.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:33.005+0000 7f632b03a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:33.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:33Z"} 2024-03-04T17:22:33.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:33Z"} 2024-03-04T17:22:33.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:33.034 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:33.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:33.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T17:22:38.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:38.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:38.035 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:38.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:38.082 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:38.131 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:38.133+0000 7fbfc9fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:38.132 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:38.134+0000 7fbfc9fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:38.154 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:38Z"} 2024-03-04T17:22:38.154 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:38Z"} 2024-03-04T17:22:38.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:38.155 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:38.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:38.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.873456_437.1 2024-03-04T17:22:43.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:43.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:43.158 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:43.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:43.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:43.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:43.292+0000 7f1cc9d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:43.292+0000 7f1cc9d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:43.324 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:43Z"} 2024-03-04T17:22:43.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:43Z"} 2024-03-04T17:22:43.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:43.324 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:43.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:43.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.860903_435.1 2024-03-04T17:22:48.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:48.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:48.327 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:48.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:48.430+0000 7fb32900f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:48.430 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:48.431+0000 7fb32900f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:48.452 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:48Z"} 2024-03-04T17:22:48.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:48Z"} 2024-03-04T17:22:48.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:48.453 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:48.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:48.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T17:22:53.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:53.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:53.454 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:53.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:53.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:53.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:53.563+0000 7f593bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:53.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:53.564+0000 7f593bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:53.585 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:53Z"} 2024-03-04T17:22:53.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:53Z"} 2024-03-04T17:22:53.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:53.586 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:53.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:53.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T17:22:58.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:58.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:22:58.588 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:22:58.635 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:22:58.635 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:22:58.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:58.699+0000 7f101734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:58.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:22:58.699+0000 7f101734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:22:58.721 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:58Z"} 2024-03-04T17:22:58.721 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:58Z"} 2024-03-04T17:22:58.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:22:58.722 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:58.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:22:58.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.951904_442.1 2024-03-04T17:23:03.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:03.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:03.724 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:03.773 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:03.773 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:03.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:03.841+0000 7f1d77812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:03.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:03.842+0000 7f1d77812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:03.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:03Z"} 2024-03-04T17:23:03.867 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:03Z"} 2024-03-04T17:23:03.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:03.867 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:03.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:03.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.934048_439.1 2024-03-04T17:23:08.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:08.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:08.869 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:08.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:08.970+0000 7f0326c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:08.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:08.971+0000 7f0326c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:08.992 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:08Z"} 2024-03-04T17:23:08.992 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:08Z"} 2024-03-04T17:23:08.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:08.993 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:08.993 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:08.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:23:13.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:13.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:13.994 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:14.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:14.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:14.094 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:14.095+0000 7fdc43331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:14.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:14.096+0000 7fdc43331800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:14.119 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:14Z"} 2024-03-04T17:23:14.120 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:14Z"} 2024-03-04T17:23:14.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:14.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:14.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:14.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.088500_457.1 2024-03-04T17:23:19.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:19.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:19.123 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:19.182 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:19.182 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:19.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:19.239+0000 7fae5fc2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:19.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:19.240+0000 7fae5fc2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:19.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:19Z"} 2024-03-04T17:23:19.265 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:19Z"} 2024-03-04T17:23:19.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:19.265 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:19.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:19.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.860903_435.1 2024-03-04T17:23:23.516 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:23:23.517+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:23:24.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:24.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:24.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:24.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:24.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:24.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:24.364+0000 7f1c38b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:24.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:24.365+0000 7f1c38b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:24.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:24Z"} 2024-03-04T17:23:24.387 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:24Z"} 2024-03-04T17:23:24.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:24.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:24.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:24.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T17:23:29.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:29.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:29.388 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:29.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:29.437 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:29.501 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:29.502+0000 7f5ad1d09800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:29.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:29.503+0000 7f5ad1d09800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:29.524 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:29Z"} 2024-03-04T17:23:29.525 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:29Z"} 2024-03-04T17:23:29.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:29.525 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:29.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:29.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.969696_444.1 2024-03-04T17:23:34.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:34.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:34.527 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:34.576 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:34.576 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:34.635 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:34.636+0000 7f5f43844800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:34.636 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:34.637+0000 7f5f43844800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:34.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:34Z"} 2024-03-04T17:23:34.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:34Z"} 2024-03-04T17:23:34.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:34.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:34.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:34.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:23:39.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:39.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:39.669 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:39.719 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:39.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:39.820+0000 7f01bf024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:39.834 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:39.833+0000 7f01bf024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:39Z"} 2024-03-04T17:23:39.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:39Z"} 2024-03-04T17:23:39.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:39.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:39.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:39.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.973308_446.1 2024-03-04T17:23:44.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:44.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:44.925 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:44.987 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:45.041 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:45.042+0000 7f68b532a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:45.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:45.043+0000 7f68b532a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:45.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:45Z"} 2024-03-04T17:23:45.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:45Z"} 2024-03-04T17:23:45.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:45.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:45.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:45.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T17:23:50.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:50.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:50.068 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:50.129 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:50.129 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:50.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:50.192+0000 7f61b1931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:50.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:50.193+0000 7f61b1931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:50.220 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.047580_518.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.047580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:50Z"} 2024-03-04T17:23:50.220 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.047580_518.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.047580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:50Z"} 2024-03-04T17:23:50.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.047580_518.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.047580Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:50.221 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:50.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.047580_518.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:50.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570680.047580_518.1 2024-03-04T17:23:55.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:55.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:23:55.223 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:23:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:23:55.275 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:23:55.337 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:55.338+0000 7f261ee38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:55.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:23:55.339+0000 7f261ee38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:23:55.363 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:55Z"} 2024-03-04T17:23:55.363 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:55Z"} 2024-03-04T17:23:55.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:23:55.364 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:55.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:23:55.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.084490_456.1 2024-03-04T17:24:00.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:00.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:00.366 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:00.414 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:00.415 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:00.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:00.464+0000 7fcf16a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:00.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:00.465+0000 7fcf16a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:00.486 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:00Z"} 2024-03-04T17:24:00.486 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:00Z"} 2024-03-04T17:24:00.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:00.486 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:00.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:00.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.997952_450.1 2024-03-04T17:24:05.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:05.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:05.489 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:05.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:05.591+0000 7fb4e4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:05.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:05.592+0000 7fb4e4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:05.613 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:05Z"} 2024-03-04T17:24:05.613 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:05Z"} 2024-03-04T17:24:05.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:05.614 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:05.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:05.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T17:24:10.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:10.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:10.616 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:10.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:10.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:10.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:10.720+0000 7f4d87c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:10.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:10.721+0000 7f4d87c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:10.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:10Z"} 2024-03-04T17:24:10.746 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:10Z"} 2024-03-04T17:24:10.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:10.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:10.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:10.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.088500_457.1 2024-03-04T17:24:15.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:15.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:15.748 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:15.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:15.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:15.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:15.849+0000 7fe09d88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:15.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:15.850+0000 7fe09d88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:15.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:15Z"} 2024-03-04T17:24:15.874 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:15Z"} 2024-03-04T17:24:15.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:15.875 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:15.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:15.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.192640_463.1 2024-03-04T17:24:20.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:20.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:20.877 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:20.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:20.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:20.982+0000 7f2df552f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:20.982 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:20.983+0000 7f2df552f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:21.009 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:20Z"} 2024-03-04T17:24:21.009 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:20Z"} 2024-03-04T17:24:21.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:21.010 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:21.010 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:21.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.071168_455.1 2024-03-04T17:24:21.397 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:24:21.398+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:24:26.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:26.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:26.011 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:26.058 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:26.058 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:26.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:26.116+0000 7fd5da130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:26.116 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:26.117+0000 7fd5da130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:26.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:26Z"} 2024-03-04T17:24:26.147 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:26Z"} 2024-03-04T17:24:26.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:26.147 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:26.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:26.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.193667_464.1 2024-03-04T17:24:31.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:31.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:31.149 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:31.195 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:31.195 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:31.259 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:31.260+0000 7f2973b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:31.259 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:31.261+0000 7f2973b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:31.287 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:31Z"} 2024-03-04T17:24:31.288 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:31Z"} 2024-03-04T17:24:31.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:31.288 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:31.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:31.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:24:36.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:36.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:36.290 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:36.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:36.337 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:36.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:36.400+0000 7fd06d72a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:36.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:36.401+0000 7fd06d72a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:36.422 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:36Z"} 2024-03-04T17:24:36.422 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:36Z"} 2024-03-04T17:24:36.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:36.422 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:36.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:36.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.160094_460.1 2024-03-04T17:24:41.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:41.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:41.425 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:41.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:41.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:41.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:41.529+0000 7fdd8fb29800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:41.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:41.530+0000 7fdd8fb29800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:41.551 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:41Z"} 2024-03-04T17:24:41.552 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:41Z"} 2024-03-04T17:24:41.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:41.552 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:41.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:41.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.155110_458.1 2024-03-04T17:24:46.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:46.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:46.554 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:46.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:46.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:46.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:46.656+0000 7f871d616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:46.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:46.656+0000 7f871d616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:46.678 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:46Z"} 2024-03-04T17:24:46.678 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:46Z"} 2024-03-04T17:24:46.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:46.678 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:46.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:46.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T17:24:51.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:51.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:51.680 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:51.728 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:51.728 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:51.798 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:51.799+0000 7fa011213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:51.798 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:51.799+0000 7fa011213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:51.827 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:51Z"} 2024-03-04T17:24:51.828 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:51Z"} 2024-03-04T17:24:51.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:51.828 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:51.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:51.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T17:24:56.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:56.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:24:56.830 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:24:56.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:24:56.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:24:56.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:56.937+0000 7fe5fee9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:56.937 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:24:56.938+0000 7fe5fee9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:24:56.969 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:56Z"} 2024-03-04T17:24:56.970 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:56Z"} 2024-03-04T17:24:56.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:24:56.970 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:56.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:24:56.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T17:25:01.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:01.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:01.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:02.025 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:02.025 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:02.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:02.085+0000 7f32ec72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:02.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:02.086+0000 7f32ec72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:02.117 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:02Z"} 2024-03-04T17:25:02.117 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:02Z"} 2024-03-04T17:25:02.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:02.117 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:02.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:02.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.994542_448.1 2024-03-04T17:25:07.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:07.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:07.119 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:07.167 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:07.167 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:07.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:07.239+0000 7fe9e272b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:07.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:07.239+0000 7fe9e272b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:07.266 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:07Z"} 2024-03-04T17:25:07.266 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:07Z"} 2024-03-04T17:25:07.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:07.266 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:07.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:07.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.015683_453.1 2024-03-04T17:25:12.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:12.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:12.269 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:12.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:12.388+0000 7f0457a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:12.388 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:12.389+0000 7f0457a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:12.424 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:12Z"} 2024-03-04T17:25:12.425 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:12Z"} 2024-03-04T17:25:12.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:12.425 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:12.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:12.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T17:25:17.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:17.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:17.427 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:17.480 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:17.481 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:17.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:17.553+0000 7fd663142800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:17.554+0000 7fd663142800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:17.582 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:17Z"} 2024-03-04T17:25:17.582 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:17Z"} 2024-03-04T17:25:17.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:17.583 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:17.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:17.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.000296_451.1 2024-03-04T17:25:22.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:22.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:22.584 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:22.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:22.667 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:22.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:22.781+0000 7f3516e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:22.781 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:22.782+0000 7f3516e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:22.803 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:22Z"} 2024-03-04T17:25:22.803 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:22Z"} 2024-03-04T17:25:22.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:22.804 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:22.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:22.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T17:25:27.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:27.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:27.806 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:27.866 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:27.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:27.941 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:27.942+0000 7f3f2612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:27.942 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:27.943+0000 7f3f2612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:27.966 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:27Z"} 2024-03-04T17:25:27.966 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:27Z"} 2024-03-04T17:25:27.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:27.966 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:27.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:27.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.084490_456.1 2024-03-04T17:25:32.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:32.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:32.968 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:33.047 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:33.145 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:33.146+0000 7f2130d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:33.146 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:33.147+0000 7f2130d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:33.170 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:33Z"} 2024-03-04T17:25:33.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:33Z"} 2024-03-04T17:25:33.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:33.171 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:33.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:33.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.973308_446.1 2024-03-04T17:25:38.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:38.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:38.173 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:38.236 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:38.237 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:38.393+0000 7f7065c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:38.397 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:38.398+0000 7f7065c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:38.435 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:38Z"} 2024-03-04T17:25:38.435 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:38Z"} 2024-03-04T17:25:38.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:38.436 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:38.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:38.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T17:25:43.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:43.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:43.437 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:43.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:43.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:43.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:43.553+0000 7f7348467800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:43.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:43.554+0000 7f7348467800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:43.586 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:43Z"} 2024-03-04T17:25:43.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:43Z"} 2024-03-04T17:25:43.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:43.586 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:43.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:43.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:25:48.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:48.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:48.588 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:48.634 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:48.634 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:48.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:48.697+0000 7fcd941a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:48.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:48.698+0000 7fcd941a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:48.722 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:48Z"} 2024-03-04T17:25:48.723 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:48Z"} 2024-03-04T17:25:48.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:48.723 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:48.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:48.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.158768_459.1 2024-03-04T17:25:53.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:53.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:53.725 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:53.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:53.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:53.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:53.827+0000 7fd17cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:53.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:53.828+0000 7fd17cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:53.852 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:53Z"} 2024-03-04T17:25:53.852 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:53Z"} 2024-03-04T17:25:53.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:53.852 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:53.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:53.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T17:25:58.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:58.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:25:58.854 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:25:58.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:25:58.900 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:25:58.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:58.963+0000 7f8573b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:58.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:25:58.964+0000 7f8573b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:25:58.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:58Z"} 2024-03-04T17:25:58.986 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:58Z"} 2024-03-04T17:25:58.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.020506_513.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.020506Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:25:58.986 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:58.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.020506_513.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:25:58.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570680.020506_513.1 2024-03-04T17:26:03.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:03.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:03.989 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:04.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:04.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:04.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:04.097+0000 7f6b4a411800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:04.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:04.098+0000 7f6b4a411800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:04.120 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:04Z"} 2024-03-04T17:26:04.121 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:04Z"} 2024-03-04T17:26:04.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:04.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:04.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:04.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:26:09.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:09.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:09.123 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:09.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:09.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:09.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:09.240+0000 7f262cc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:09.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:09.241+0000 7f262cc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:09.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:09Z"} 2024-03-04T17:26:09.264 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:09Z"} 2024-03-04T17:26:09.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:09.265 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:09.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:09.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T17:26:14.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:14.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:14.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:14.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:14.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:14.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:14.376+0000 7fd669142800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:14.377+0000 7fd669142800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:14.399 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:14Z"} 2024-03-04T17:26:14.399 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:14Z"} 2024-03-04T17:26:14.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:14.400 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:14.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:14.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.994542_448.1 2024-03-04T17:26:19.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:19.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:19.402 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:19.453 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:19.453 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:19.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:19.537+0000 7ff1f5c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:19.537 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:19.538+0000 7ff1f5c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:19.566 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:19Z"} 2024-03-04T17:26:19.567 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:19Z"} 2024-03-04T17:26:19.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:19.567 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:19.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:19.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.996984_449.1 2024-03-04T17:26:24.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:24.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:24.569 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:24.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:24.674+0000 7fa96dc13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:24.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:24.675+0000 7fa96dc13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:24.711 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:24Z"} 2024-03-04T17:26:24.711 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:24Z"} 2024-03-04T17:26:24.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:24.712 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:24.712 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:24.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.996984_449.1 2024-03-04T17:26:29.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:29.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:29.714 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:29.761 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:29.761 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:29.836 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:29.837+0000 7f2863e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:29.837 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:29.837+0000 7f2863e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:29Z"} 2024-03-04T17:26:29.864 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:29Z"} 2024-03-04T17:26:29.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:29.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:29.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:29.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.015683_453.1 2024-03-04T17:26:34.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:34.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:34.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:34.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:34.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:34.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:34.968+0000 7f749492e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:34.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:34.969+0000 7f749492e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:34.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:34Z"} 2024-03-04T17:26:34.994 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:34Z"} 2024-03-04T17:26:34.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:34.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:34.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:34.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570988.608330_559.1 2024-03-04T17:26:39.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:39.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:39.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:40.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:40.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:40.114+0000 7f0857d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:40.115+0000 7f0857d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:40.142 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:40Z"} 2024-03-04T17:26:40.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:40Z"} 2024-03-04T17:26:40.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:40.143 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:40.143 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:40.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T17:26:45.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:45.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:45.145 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:45.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:45.194 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:45.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:45.243+0000 7fc479e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:45.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:45.244+0000 7fc479e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:45.275 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:45Z"} 2024-03-04T17:26:45.275 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:45Z"} 2024-03-04T17:26:45.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:45.276 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:45.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:45.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.158768_459.1 2024-03-04T17:26:50.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:50.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:50.278 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:50.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:50.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:50.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:50.389+0000 7efd24b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:50.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:50.390+0000 7efd24b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:50.415 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:50Z"} 2024-03-04T17:26:50.415 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:50Z"} 2024-03-04T17:26:50.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:50.416 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:50.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:50.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T17:26:55.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:55.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:26:55.418 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:26:55.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:26:55.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:26:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:55.536+0000 7f9a7b02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:26:55.537+0000 7f9a7b02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:26:55.561 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:55Z"} 2024-03-04T17:26:55.561 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:55Z"} 2024-03-04T17:26:55.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:26:55.562 DEBUG:rgw_multi.tests:log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:55.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:26:55.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.215333_466.1 2024-03-04T17:27:00.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-03-04T17:27:00.566 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-67 2024-03-04T17:27:00.567 DEBUG:boto:path=/sbnvju-67/ 2024-03-04T17:27:00.567 DEBUG:boto:auth_path=/sbnvju-67/ 2024-03-04T17:27:00.567 DEBUG:boto:Method: PUT 2024-03-04T17:27:00.567 DEBUG:boto:Path: /sbnvju-67/ 2024-03-04T17:27:00.567 DEBUG:boto:Data: 2024-03-04T17:27:00.567 DEBUG:boto:Headers: {} 2024-03-04T17:27:00.567 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:27:00.567 DEBUG:boto:Port: 8000 2024-03-04T17:27:00.567 DEBUG:boto:Params: {} 2024-03-04T17:27:00.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:27:00.568 DEBUG:boto:Token: None 2024-03-04T17:27:00.568 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:27:00 GMT /sbnvju-67/ 2024-03-04T17:27:00.570 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:v4xeQS2ZcuhEdBwHABazzpCLr+I= 2024-03-04T17:27:00.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v4xeQS2ZcuhEdBwHABazzpCLr+I=', 'Content-Length': '0'} 2024-03-04T17:27:00.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b41eeddde4b6640-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:27:00.593 DEBUG:boto:path=/sbnvju-67/testobj-sse-c 2024-03-04T17:27:00.594 DEBUG:boto:auth_path=/sbnvju-67/testobj-sse-c 2024-03-04T17:27:00.594 DEBUG:boto:Method: PUT 2024-03-04T17:27:00.594 DEBUG:boto:Path: /sbnvju-67/testobj-sse-c 2024-03-04T17:27:00.594 DEBUG:boto:Data: 2024-03-04T17:27:00.594 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T17:27:00.594 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:27:00.594 DEBUG:boto:Port: 8000 2024-03-04T17:27:00.594 DEBUG:boto:Params: {} 2024-03-04T17:27:00.594 DEBUG:boto:Token: None 2024-03-04T17:27:00.594 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 17:27:00 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /sbnvju-67/testobj-sse-c 2024-03-04T17:27:00.595 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:QKS4IhAwJ/BtpozTWRLklJkVuZE= 2024-03-04T17:27:00.595 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QKS4IhAwJ/BtpozTWRLklJkVuZE='} 2024-03-04T17:27:00.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000060c1f752b9a260d6-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:27:00.603 DEBUG:boto:path=/sbnvju-67/testobj-sse-kms 2024-03-04T17:27:00.603 DEBUG:boto:auth_path=/sbnvju-67/testobj-sse-kms 2024-03-04T17:27:00.604 DEBUG:boto:Method: PUT 2024-03-04T17:27:00.604 DEBUG:boto:Path: /sbnvju-67/testobj-sse-kms 2024-03-04T17:27:00.604 DEBUG:boto:Data: 2024-03-04T17:27:00.604 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T17:27:00.604 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:27:00.604 DEBUG:boto:Port: 8000 2024-03-04T17:27:00.604 DEBUG:boto:Params: {} 2024-03-04T17:27:00.604 DEBUG:boto:Token: None 2024-03-04T17:27:00.604 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 04 Mar 2024 17:27:00 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /sbnvju-67/testobj-sse-kms 2024-03-04T17:27:00.605 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PZd62gFyi6c+vVteVmgYLlCe3pI= 2024-03-04T17:27:00.605 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PZd62gFyi6c+vVteVmgYLlCe3pI='} 2024-03-04T17:27:00.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6c2Judmp1LTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000e0d8f2cf02f9d59b-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:27:00.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:00.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:00.635 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:00.746 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:27:00.768 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709573220.591929_891.1","last_update":"2024-03-04T17:27:00.591929Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:27:00.769 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:00.769 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:27:00.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:00.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:00.769 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:00.844 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:00.844 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:00.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:00.920+0000 7f155de2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:00.920 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:00.921+0000 7f155de2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:00.963 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:00Z"} 2024-03-04T17:27:00.963 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:00Z"} 2024-03-04T17:27:00.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:00.964 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:00.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:00.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:27:05.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:05.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:05.965 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:06.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:06.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:06.063 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:06.063+0000 7fee36333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:06.064 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:06.064+0000 7fee36333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:06.088 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:06Z"} 2024-03-04T17:27:06.088 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:06Z"} 2024-03-04T17:27:06.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:06.089 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:06.089 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:06.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:27:11.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:11.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:11.091 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:11.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:11.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:11.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:11.213+0000 7f2a0452f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:11.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:11.214+0000 7f2a0452f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:11.237 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:11Z"} 2024-03-04T17:27:11.238 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:11Z"} 2024-03-04T17:27:11.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:11.238 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:11.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:11.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T17:27:16.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:16.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:16.241 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:16.288 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:16.288 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:16.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:16.345+0000 7f022702f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:16.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:16.346+0000 7f022702f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:16.373 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:16Z"} 2024-03-04T17:27:16.373 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:16Z"} 2024-03-04T17:27:16.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:16.374 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:16.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:16.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:27:21.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:21.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:21.376 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:21.423 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:21.423 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:21.485 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:21.484+0000 7fe45522f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:21.485 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:21.485+0000 7fe45522f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:21.511 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:21Z"} 2024-03-04T17:27:21.512 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:21Z"} 2024-03-04T17:27:21.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:21.512 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:21.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:21.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.193667_464.1 2024-03-04T17:27:26.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:26.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:26.513 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:26.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:26.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:26.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:26.621+0000 7f3610413800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:26.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:26.622+0000 7f3610413800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:26.650 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:26Z"} 2024-03-04T17:27:26.650 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:26Z"} 2024-03-04T17:27:26.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.078213_520.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.078213Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:26.650 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:26.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.078213_520.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:26.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.078213_520.1 2024-03-04T17:27:31.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:31.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:31.652 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:31.699 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:31.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:31.754+0000 7f917c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:31.755+0000 7f917c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:31.777 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:31Z"} 2024-03-04T17:27:31.777 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:31Z"} 2024-03-04T17:27:31.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:31.778 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:31.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:31.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:27:36.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:36.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:36.780 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:36.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:36.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:36.876 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:36.876+0000 7fd3f994b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:36.877 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:36.877+0000 7fd3f994b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:36.900 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:36Z"} 2024-03-04T17:27:36.900 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:36Z"} 2024-03-04T17:27:36.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:36.901 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:36.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:36.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T17:27:41.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:41.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:41.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:41.951 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:41.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:42.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:42.023+0000 7fa8fea12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:42.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:42.024+0000 7fa8fea12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:42.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:42Z"} 2024-03-04T17:27:42.051 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:42Z"} 2024-03-04T17:27:42.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:42.052 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:42.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:42.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:27:47.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:47.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:47.056 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:47.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:47.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:47.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:47.151+0000 7fa10c830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:47.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:47.152+0000 7fa10c830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:47.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:47Z"} 2024-03-04T17:27:47.183 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:47Z"} 2024-03-04T17:27:47.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:47.184 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:47.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:47.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T17:27:52.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:52.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:52.186 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:52.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:52.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:52.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:52.300+0000 7f7617931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:52.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:52.301+0000 7f7617931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:52.327 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:52Z"} 2024-03-04T17:27:52.327 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:52Z"} 2024-03-04T17:27:52.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:52.327 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:52.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:52.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.088500_457.1 2024-03-04T17:27:57.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:57.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:27:57.329 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:27:57.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:27:57.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:27:57.440 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:57.441+0000 7ff57c814800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:57.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:27:57.442+0000 7ff57c814800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:27:57.467 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:57Z"} 2024-03-04T17:27:57.467 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:57Z"} 2024-03-04T17:27:57.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:27:57.468 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:57.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:27:57.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T17:28:02.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:02.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:02.470 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:02.520 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:02.521 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:02.583 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:02.583+0000 7f87f1632800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:02.584 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:02.584+0000 7f87f1632800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:02.616 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:02Z"} 2024-03-04T17:28:02.617 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:02Z"} 2024-03-04T17:28:02.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:02.617 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:02.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:02.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T17:28:07.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:07.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:07.619 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:07.672 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:07.672 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:07.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:07.733+0000 7f07828c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:07.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:07.734+0000 7f07828c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:07.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:07Z"} 2024-03-04T17:28:07.758 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:07Z"} 2024-03-04T17:28:07.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:07.758 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:07.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:07.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T17:28:12.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:12.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:12.759 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:12.814 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:12.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:12.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:12.869+0000 7fce18213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:12.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:12.870+0000 7fce18213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:12.895 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:12Z"} 2024-03-04T17:28:12.895 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:12Z"} 2024-03-04T17:28:12.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:12.895 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:12.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:12.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T17:28:17.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:17.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:17.898 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:17.949 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:17.949 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:18.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:18.008+0000 7ffb8508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:18.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:18.009+0000 7ffb8508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:18.034 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:18Z"} 2024-03-04T17:28:18.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:18Z"} 2024-03-04T17:28:18.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:18.035 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:18.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:18.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T17:28:23.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:23.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:23.037 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:23.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:23.152+0000 7f701ed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:23.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:23.152+0000 7f701ed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:23.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:23Z"} 2024-03-04T17:28:23.179 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:23Z"} 2024-03-04T17:28:23.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:23.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:23.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:23.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709571978.116657_707.1 2024-03-04T17:28:28.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:28.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:28.181 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:28.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:28.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:28.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:28.291+0000 7fcbb8e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:28.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:28.292+0000 7fcbb8e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:28.316 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:28Z"} 2024-03-04T17:28:28.316 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:28Z"} 2024-03-04T17:28:28.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:28.317 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:28.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:28.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T17:28:33.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:33.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:33.318 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:33.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:33.437+0000 7fe375211800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:33.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:33.438+0000 7fe375211800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:33.472 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.021888_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.021888Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:33Z"} 2024-03-04T17:28:33.472 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.021888_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.021888Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:33Z"} 2024-03-04T17:28:33.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.021888_514.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.021888Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:33.473 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:33.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.021888_514.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:33.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.021888_514.1 2024-03-04T17:28:38.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:38.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:38.475 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:38.529 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:38.529 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:38.581 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:38.581+0000 7f4a47613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:38.582 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:38.582+0000 7f4a47613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:38.610 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:38Z"} 2024-03-04T17:28:38.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:38Z"} 2024-03-04T17:28:38.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:38.611 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:38.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:38.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:28:43.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:43.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:43.614 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:43.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:43.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:43.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:43.748+0000 7f0d46524800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:43.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:43.749+0000 7f0d46524800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:43.783 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:43Z"} 2024-03-04T17:28:43.784 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:43Z"} 2024-03-04T17:28:43.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:43.784 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:43.784 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:43.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.972318_445.1 2024-03-04T17:28:48.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:48.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:48.786 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:48.833 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:48.833 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:48.884 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:48.884+0000 7f9fea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:48.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:48.885+0000 7f9fea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:48.907 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:48Z"} 2024-03-04T17:28:48.908 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:48Z"} 2024-03-04T17:28:48.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:48.908 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:48.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:48.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.192640_463.1 2024-03-04T17:28:53.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:53.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:53.911 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:53.960 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:53.960 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:54.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:54.023+0000 7f1ad729d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:54.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:54.024+0000 7f1ad729d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:54.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:54Z"} 2024-03-04T17:28:54.057 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:54Z"} 2024-03-04T17:28:54.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:54.057 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:54.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:54.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T17:28:59.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:59.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:28:59.059 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:28:59.115 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:28:59.115 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:28:59.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:59.230+0000 7f2371528800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:59.232 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:28:59.231+0000 7f2371528800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:28:59.258 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:59Z"} 2024-03-04T17:28:59.258 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:59Z"} 2024-03-04T17:28:59.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:28:59.259 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:59.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:28:59.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T17:29:04.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:04.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:04.260 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:04.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:04.314 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:04.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:04.472+0000 7fed12213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:04.474 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:04.473+0000 7fed12213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:04.537 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:04Z"} 2024-03-04T17:29:04.537 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:04Z"} 2024-03-04T17:29:04.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:04.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:04.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:04.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:29:09.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:09.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:09.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:09.586 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:09.586 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:09.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:09.655+0000 7fb6a5f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:09.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:09.656+0000 7fb6a5f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:09.679 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:09Z"} 2024-03-04T17:29:09.680 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:09Z"} 2024-03-04T17:29:09.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:09.680 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:09.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:09.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:29:14.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:14.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:14.682 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:14.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:14.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:14.782 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:14.781+0000 7f5fd0c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:14.783 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:14.782+0000 7f5fd0c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:14.805 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:14Z"} 2024-03-04T17:29:14.805 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:14Z"} 2024-03-04T17:29:14.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:14.806 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:14.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:14.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:29:19.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:19.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:19.808 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:19.855 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:19.855 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:19.916 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:19.915+0000 7fb7e9630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:19.917 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:19.916+0000 7fb7e9630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:19.942 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:19Z"} 2024-03-04T17:29:19.943 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:19Z"} 2024-03-04T17:29:19.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:19.943 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:19.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:19.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.064903_454.1 2024-03-04T17:29:24.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:24.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:24.945 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:24.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:24.991 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:25.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:25.056+0000 7f6065244800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:25.057 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:25.057+0000 7f6065244800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:25.085 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:25Z"} 2024-03-04T17:29:25.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:25Z"} 2024-03-04T17:29:25.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:25.085 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:25.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:25.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.189913_462.1 2024-03-04T17:29:30.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:30.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:30.093 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:30.142 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:30.142 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:30.197 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:30.196+0000 7fd337215800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:30.197 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:30.197+0000 7fd337215800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:30.228 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.025016_515.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.025016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:30Z"} 2024-03-04T17:29:30.228 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.025016_515.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.025016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:30Z"} 2024-03-04T17:29:30.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.025016_515.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.025016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:30.228 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:30.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.025016_515.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:30.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.025016_515.1 2024-03-04T17:29:35.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:35.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:35.230 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:35.279 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:35.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:35.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:35.331+0000 7fbbd5530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:35.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:35.332+0000 7fbbd5530800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:35.354 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:35Z"} 2024-03-04T17:29:35.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:35Z"} 2024-03-04T17:29:35.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:35.355 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:35.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:35.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T17:29:39.019 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:29:39.018+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:29:40.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:40.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:40.357 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:40.405 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:40.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:40.453+0000 7f65b2d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:40.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:40.454+0000 7f65b2d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:40.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:40Z"} 2024-03-04T17:29:40.479 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:40Z"} 2024-03-04T17:29:40.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:40.479 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:40.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:40.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.215333_466.1 2024-03-04T17:29:45.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:45.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:45.481 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:45.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:45.529 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:45.575 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:45.575+0000 7f2eaa6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:45.576 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:45.575+0000 7f2eaa6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:45.598 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:45Z"} 2024-03-04T17:29:45.599 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:45Z"} 2024-03-04T17:29:45.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:45.599 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:45.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:45.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.064903_454.1 2024-03-04T17:29:50.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:50.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:50.601 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:50.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:50.652 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:50.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:50.721+0000 7f9010083800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:50.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:50.722+0000 7f9010083800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:50.771 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:50Z"} 2024-03-04T17:29:50.771 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:50Z"} 2024-03-04T17:29:50.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:50.772 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:50.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:50.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T17:29:55.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:55.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:29:55.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:29:55.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:29:55.823 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:29:55.891 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:55.890+0000 7f45c1f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:55.892 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:29:55.891+0000 7f45c1f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:29:55.913 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:55Z"} 2024-03-04T17:29:55.914 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:55Z"} 2024-03-04T17:29:55.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:29:55.914 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:55.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:29:55.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T17:30:00.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:00.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:00.916 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:00.966 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:00.966 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:01.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:01.025+0000 7fb784520800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:01.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:01.026+0000 7fb784520800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:01.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:01Z"} 2024-03-04T17:30:01.051 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:01Z"} 2024-03-04T17:30:01.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:01.051 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:01.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:01.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T17:30:06.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:06.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:06.053 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:06.102 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:06.102 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:06.165+0000 7f994f212800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:06.166+0000 7f994f212800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:06.197 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:06Z"} 2024-03-04T17:30:06.198 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:06Z"} 2024-03-04T17:30:06.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:06.198 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:06.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:06.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:30:11.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:11.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:11.201 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:11.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:11.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:11.312+0000 7f227382e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:11.313+0000 7f227382e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:11.337 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:11Z"} 2024-03-04T17:30:11.338 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:11Z"} 2024-03-04T17:30:11.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:11.338 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:11.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:11.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T17:30:16.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:16.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:16.340 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:16.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:16.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:16.450 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:16.449+0000 7f4306511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:16.451 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:16.450+0000 7f4306511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:16.475 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:16Z"} 2024-03-04T17:30:16.476 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:16Z"} 2024-03-04T17:30:16.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:16.476 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:16.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:16.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570988.616944_562.1 2024-03-04T17:30:21.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:21.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:21.478 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:21.530 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:21.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:21.574+0000 7f04fba24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:21.575 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:21.575+0000 7f04fba24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:21.600 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:21Z"} 2024-03-04T17:30:21.601 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:21Z"} 2024-03-04T17:30:21.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:21.601 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:21.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:21.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:30:26.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:26.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:26.603 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:26.653 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:26.653 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:26.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:26.718+0000 7f9e44b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:26.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:26.719+0000 7f9e44b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:26.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:26Z"} 2024-03-04T17:30:26.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:26Z"} 2024-03-04T17:30:26.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:26.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:26.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:26.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T17:30:31.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:31.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:31.750 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:31.800 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:31.801 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:31.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:31.865+0000 7fba28f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:31.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:31.866+0000 7fba28f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:31.898 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:31Z"} 2024-03-04T17:30:31.898 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:31Z"} 2024-03-04T17:30:31.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:31.899 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:31.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:31.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:30:36.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:36.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:36.900 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:36.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:36.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:37.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:37.018+0000 7f7947416800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:37.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:37.019+0000 7f7947416800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:37Z"} 2024-03-04T17:30:37.052 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:37Z"} 2024-03-04T17:30:37.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:37.052 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:37.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:37.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.158768_459.1 2024-03-04T17:30:42.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:42.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:42.054 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:42.107 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:42.107 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:42.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:42.176+0000 7f6a16613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:42.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:42.182+0000 7f6a16613800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:42.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:42Z"} 2024-03-04T17:30:42.283 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:42Z"} 2024-03-04T17:30:42.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:42.283 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:42.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:42.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:30:47.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:47.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:47.284 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:47.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:47.451+0000 7fbcc512e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:47.456 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:47.452+0000 7fbcc512e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:47.481 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:47Z"} 2024-03-04T17:30:47.481 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:47Z"} 2024-03-04T17:30:47.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:47.482 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:47.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:47.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:30:52.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:52.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:52.484 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:52.530 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:52.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:52.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:52.596+0000 7f144ad3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:52.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:52.597+0000 7f144ad3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:52.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:52Z"} 2024-03-04T17:30:52.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:52Z"} 2024-03-04T17:30:52.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:52.629 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:52.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:52.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:30:57.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:57.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:30:57.631 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:30:57.679 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:30:57.679 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:30:57.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:57.744+0000 7fd88e28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:57.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:30:57.745+0000 7fd88e28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:30:57.772 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:57Z"} 2024-03-04T17:30:57.772 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:57Z"} 2024-03-04T17:30:57.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:30:57.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:57.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:30:57.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T17:31:02.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:02.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:02.775 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:02.821 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:02.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:02.873 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:02.872+0000 7f93f5711800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:02.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:02.873+0000 7f93f5711800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:02.908 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:02Z"} 2024-03-04T17:31:02.908 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:02Z"} 2024-03-04T17:31:02.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:02.909 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:02.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:02.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T17:31:07.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:07.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:07.910 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:07.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:07.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:08.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:08.018+0000 7f4f75424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:08.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:08.019+0000 7f4f75424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:08.047 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:08Z"} 2024-03-04T17:31:08.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:08Z"} 2024-03-04T17:31:08.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:08.048 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:08.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:08.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T17:31:13.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:13.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:13.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:13.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:13.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:13.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:13.160+0000 7f1523624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:13.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:13.160+0000 7f1523624800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:13.189 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:13Z"} 2024-03-04T17:31:13.189 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:13Z"} 2024-03-04T17:31:13.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:13.190 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:13.190 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:13.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T17:31:18.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:18.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:18.194 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:18.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:18.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:18.304+0000 7fd47ea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:18.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:18.306+0000 7fd47ea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:18.340 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:18Z"} 2024-03-04T17:31:18.340 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:18Z"} 2024-03-04T17:31:18.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:18.341 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:18.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:18.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T17:31:23.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:23.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:23.342 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:23.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:23.393 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:23.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:23.442+0000 7f0395b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:23.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:23.443+0000 7f0395b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:23.466 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:23Z"} 2024-03-04T17:31:23.467 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:23Z"} 2024-03-04T17:31:23.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:23.467 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:23.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:23.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:31:28.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:28.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:28.469 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:28.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:28.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:28.583 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:28.582+0000 7f5feb52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:28.583 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:28.582+0000 7f5feb52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:28.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:28Z"} 2024-03-04T17:31:28.609 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:28Z"} 2024-03-04T17:31:28.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:28.610 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:28.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:28.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T17:31:33.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:33.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:33.611 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:33.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:33.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:33.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:33.712+0000 7f4572939800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:33.714 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:33.713+0000 7f4572939800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:33.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:33Z"} 2024-03-04T17:31:33.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:33Z"} 2024-03-04T17:31:33.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:33.738 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:33.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:33.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T17:31:38.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:38.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:38.739 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:38.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:38.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:38.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:38.850+0000 7f1d3a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:38.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:38.851+0000 7f1d3a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:38.876 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:38Z"} 2024-03-04T17:31:38.876 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:38Z"} 2024-03-04T17:31:38.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.020506_513.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.020506Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:38.877 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:38.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.020506_513.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:38.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.020506_513.1 2024-03-04T17:31:43.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:43.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:43.879 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:43.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:43.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:43.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:43.986+0000 7f76d4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:43.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:43.987+0000 7f76d4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:44.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:43Z"} 2024-03-04T17:31:44.016 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:43Z"} 2024-03-04T17:31:44.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:44.017 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:44.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:44.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T17:31:49.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:49.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:49.019 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:49.075 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:49.075 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:49.147 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:49.146+0000 7effefb2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:49.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:49.147+0000 7effefb2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:49.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:49Z"} 2024-03-04T17:31:49.180 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:49Z"} 2024-03-04T17:31:49.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:49.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:49.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:49.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:31:54.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:54.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:54.182 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:54.238 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:54.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:54.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:54.293+0000 7f3ce9616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:54.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:54.294+0000 7f3ce9616800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:54.319 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:54Z"} 2024-03-04T17:31:54.319 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:54Z"} 2024-03-04T17:31:54.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:54.320 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:54.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:54.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T17:31:59.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:59.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:31:59.322 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:31:59.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:31:59.370 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:31:59.437 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:59.436+0000 7f85d6e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:59.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:31:59.437+0000 7f85d6e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:31:59.463 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:59Z"} 2024-03-04T17:31:59.464 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:59Z"} 2024-03-04T17:31:59.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:31:59.464 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:59.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:31:59.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T17:32:04.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:04.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:04.467 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:04.520 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:04.520 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:04.593 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:04.591+0000 7f448a311800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:04.593 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:04.592+0000 7f448a311800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:04.618 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:04Z"} 2024-03-04T17:32:04.618 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:04Z"} 2024-03-04T17:32:04.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:04.619 DEBUG:rgw_multi.tests:log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:04.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:04.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T17:32:09.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2024-03-04T17:32:09.623 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-68 2024-03-04T17:32:09.623 DEBUG:boto:path=/sbnvju-68/ 2024-03-04T17:32:09.623 DEBUG:boto:auth_path=/sbnvju-68/ 2024-03-04T17:32:09.623 DEBUG:boto:Method: PUT 2024-03-04T17:32:09.623 DEBUG:boto:Path: /sbnvju-68/ 2024-03-04T17:32:09.623 DEBUG:boto:Data: 2024-03-04T17:32:09.624 DEBUG:boto:Headers: {} 2024-03-04T17:32:09.624 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.624 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.624 DEBUG:boto:Params: {} 2024-03-04T17:32:09.624 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:32:09.625 DEBUG:boto:Token: None 2024-03-04T17:32:09.625 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/ 2024-03-04T17:32:09.625 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:KEpGKJ9f2J3R2kBJ6mDUcOcVQwU= 2024-03-04T17:32:09.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KEpGKJ9f2J3R2kBJ6mDUcOcVQwU=', 'Content-Length': '0'} 2024-03-04T17:32:09.664 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5715aa3b4298ad5-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.665 DEBUG:boto:path=/sbnvju-68/ 2024-03-04T17:32:09.665 DEBUG:boto:auth_path=/sbnvju-68/ 2024-03-04T17:32:09.665 DEBUG:boto:Method: HEAD 2024-03-04T17:32:09.665 DEBUG:boto:Path: /sbnvju-68/ 2024-03-04T17:32:09.665 DEBUG:boto:Data: 2024-03-04T17:32:09.666 DEBUG:boto:Headers: {} 2024-03-04T17:32:09.666 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.666 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.666 DEBUG:boto:Params: {} 2024-03-04T17:32:09.666 DEBUG:boto:Token: None 2024-03-04T17:32:09.666 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/ 2024-03-04T17:32:09.666 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T17:32:09.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T17:32:09.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086fd00587c51b916-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.669 DEBUG:boto:path=/sbnvju-68/a 2024-03-04T17:32:09.669 DEBUG:boto:auth_path=/sbnvju-68/a 2024-03-04T17:32:09.669 DEBUG:boto:Method: PUT 2024-03-04T17:32:09.669 DEBUG:boto:Path: /sbnvju-68/a 2024-03-04T17:32:09.669 DEBUG:boto:Data: 2024-03-04T17:32:09.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:32:09.670 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.670 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.670 DEBUG:boto:Params: {} 2024-03-04T17:32:09.670 DEBUG:boto:Token: None 2024-03-04T17:32:09.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/a 2024-03-04T17:32:09.670 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:2CH7G/TWEEbTrwBmPy6bDZ+3Hqg= 2024-03-04T17:32:09.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2CH7G/TWEEbTrwBmPy6bDZ+3Hqg='} 2024-03-04T17:32:09.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049c7f6aea709ec37-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.678 DEBUG:boto:path=/sbnvju-68/ 2024-03-04T17:32:09.678 DEBUG:boto:auth_path=/sbnvju-68/ 2024-03-04T17:32:09.678 DEBUG:boto:Method: HEAD 2024-03-04T17:32:09.678 DEBUG:boto:Path: /sbnvju-68/ 2024-03-04T17:32:09.678 DEBUG:boto:Data: 2024-03-04T17:32:09.678 DEBUG:boto:Headers: {} 2024-03-04T17:32:09.679 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.679 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.679 DEBUG:boto:Params: {} 2024-03-04T17:32:09.679 DEBUG:boto:Token: None 2024-03-04T17:32:09.679 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/ 2024-03-04T17:32:09.679 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T17:32:09.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T17:32:09.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e60f5f2261be0d-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.682 DEBUG:boto:path=/sbnvju-68/b 2024-03-04T17:32:09.682 DEBUG:boto:auth_path=/sbnvju-68/b 2024-03-04T17:32:09.682 DEBUG:boto:Method: PUT 2024-03-04T17:32:09.682 DEBUG:boto:Path: /sbnvju-68/b 2024-03-04T17:32:09.682 DEBUG:boto:Data: 2024-03-04T17:32:09.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:32:09.682 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.683 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.683 DEBUG:boto:Params: {} 2024-03-04T17:32:09.683 DEBUG:boto:Token: None 2024-03-04T17:32:09.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/b 2024-03-04T17:32:09.683 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Q/x6FuP9XPA0S1uOv2eoQiuHxYQ= 2024-03-04T17:32:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q/x6FuP9XPA0S1uOv2eoQiuHxYQ='} 2024-03-04T17:32:09.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6db21227d4f0439-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.732 DEBUG:boto:path=/sbnvju-68/ 2024-03-04T17:32:09.733 DEBUG:boto:auth_path=/sbnvju-68/ 2024-03-04T17:32:09.733 DEBUG:boto:Method: HEAD 2024-03-04T17:32:09.733 DEBUG:boto:Path: /sbnvju-68/ 2024-03-04T17:32:09.733 DEBUG:boto:Data: 2024-03-04T17:32:09.733 DEBUG:boto:Headers: {} 2024-03-04T17:32:09.733 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.733 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.733 DEBUG:boto:Params: {} 2024-03-04T17:32:09.733 DEBUG:boto:Token: None 2024-03-04T17:32:09.733 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/ 2024-03-04T17:32:09.734 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T17:32:09.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T17:32:09.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046d902ba872188ad-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.736 DEBUG:boto:path=/sbnvju-68/c 2024-03-04T17:32:09.736 DEBUG:boto:auth_path=/sbnvju-68/c 2024-03-04T17:32:09.736 DEBUG:boto:Method: PUT 2024-03-04T17:32:09.737 DEBUG:boto:Path: /sbnvju-68/c 2024-03-04T17:32:09.737 DEBUG:boto:Data: 2024-03-04T17:32:09.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:32:09.737 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.737 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.737 DEBUG:boto:Params: {} 2024-03-04T17:32:09.737 DEBUG:boto:Token: None 2024-03-04T17:32:09.737 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/c 2024-03-04T17:32:09.737 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+9Nw/ZyT/FbfCvOuZH7ReimYLjA= 2024-03-04T17:32:09.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+9Nw/ZyT/FbfCvOuZH7ReimYLjA='} 2024-03-04T17:32:09.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006881186644cef2eb-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.746 DEBUG:boto:path=/sbnvju-68/ 2024-03-04T17:32:09.747 DEBUG:boto:auth_path=/sbnvju-68/ 2024-03-04T17:32:09.747 DEBUG:boto:Method: HEAD 2024-03-04T17:32:09.747 DEBUG:boto:Path: /sbnvju-68/ 2024-03-04T17:32:09.747 DEBUG:boto:Data: 2024-03-04T17:32:09.747 DEBUG:boto:Headers: {} 2024-03-04T17:32:09.747 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.747 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.747 DEBUG:boto:Params: {} 2024-03-04T17:32:09.747 DEBUG:boto:Token: None 2024-03-04T17:32:09.748 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/ 2024-03-04T17:32:09.748 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T17:32:09.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T17:32:09.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bff49e35a9c7d48-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.752 DEBUG:boto:path=/sbnvju-68/d 2024-03-04T17:32:09.752 DEBUG:boto:auth_path=/sbnvju-68/d 2024-03-04T17:32:09.752 DEBUG:boto:Method: PUT 2024-03-04T17:32:09.752 DEBUG:boto:Path: /sbnvju-68/d 2024-03-04T17:32:09.752 DEBUG:boto:Data: 2024-03-04T17:32:09.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:32:09.753 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:32:09.753 DEBUG:boto:Port: 8000 2024-03-04T17:32:09.753 DEBUG:boto:Params: {} 2024-03-04T17:32:09.753 DEBUG:boto:Token: None 2024-03-04T17:32:09.753 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:32:09 GMT /sbnvju-68/d 2024-03-04T17:32:09.753 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:3lXHfAoomqFZ3WzTkeh6aF5RkRA= 2024-03-04T17:32:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3lXHfAoomqFZ3WzTkeh6aF5RkRA='} 2024-03-04T17:32:09.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006951e6b1dabb79ba-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:32:09.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:09.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:09.801 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:09.904 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:32:09.924 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709573529.662099_919.1","last_update":"2024-03-04T17:32:09.662099Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:32:09.924 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:09.924 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:32:09.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:09.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:09.925 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:09.972 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:09.972 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:10.019+0000 7f483f3a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:10.020+0000 7f483f3a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:10.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:10Z"} 2024-03-04T17:32:10.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:10Z"} 2024-03-04T17:32:10.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:10.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:10.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:10.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T17:32:15.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:15.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:15.051 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:15.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:15.099 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:15.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:15.153+0000 7fd130c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:15.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:15.153+0000 7fd130c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:15.184 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:15Z"} 2024-03-04T17:32:15.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:15Z"} 2024-03-04T17:32:15.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:15.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:15.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:15.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T17:32:20.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:20.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:20.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:20.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:20.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:20.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:20.302+0000 7f02e8c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:20.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:20.303+0000 7f02e8c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:20.327 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:20Z"} 2024-03-04T17:32:20.327 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:20Z"} 2024-03-04T17:32:20.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:20.328 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:20.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:20.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:32:21.373 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:32:21.369+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:32:25.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:25.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:25.330 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:25.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:25.433+0000 7fedb8812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:25.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:25.434+0000 7fedb8812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:25.459 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:25Z"} 2024-03-04T17:32:25.459 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:25Z"} 2024-03-04T17:32:25.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:25.460 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:25.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:25.460 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:32:30.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:30.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:30.461 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:30.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:30.562+0000 7f1aa9030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:30.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:30.563+0000 7f1aa9030800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:30.586 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:30Z"} 2024-03-04T17:32:30.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:30Z"} 2024-03-04T17:32:30.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:30.587 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:30.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:30.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T17:32:35.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:35.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:35.589 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:35.706 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:35.706+0000 7f178310f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:35.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:35.707+0000 7f178310f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:35.735 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:35Z"} 2024-03-04T17:32:35.735 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:35Z"} 2024-03-04T17:32:35.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:35.736 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:35.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:35.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T17:32:40.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:40.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:40.737 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:40.783 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:40.831 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:40.829+0000 7f4d30a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:40.832 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:40.830+0000 7f4d30a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:40.853 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:40Z"} 2024-03-04T17:32:40.854 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:40Z"} 2024-03-04T17:32:40.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:40.854 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:40.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:40.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:32:45.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:45.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:45.856 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:45.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:45.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:45.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:45.958+0000 7f25451a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:45.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:45.959+0000 7f25451a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:45.984 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:45Z"} 2024-03-04T17:32:45.985 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:45Z"} 2024-03-04T17:32:45.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:45.985 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:45.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:45.985 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:32:50.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:50.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:50.987 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:51.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:51.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:51.080+0000 7f2459a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:51.081+0000 7f2459a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:51.105 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:51Z"} 2024-03-04T17:32:51.105 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:51Z"} 2024-03-04T17:32:51.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:51.106 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:51.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:51.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T17:32:56.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:56.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:32:56.107 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:32:56.154 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:32:56.155 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:32:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:56.215+0000 7efd4453c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:32:56.216+0000 7efd4453c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:32:56.247 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:56Z"} 2024-03-04T17:32:56.247 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:56Z"} 2024-03-04T17:32:56.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:32:56.248 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:56.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:32:56.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T17:33:01.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:01.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:01.249 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:01.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:01.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:01.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:01.361+0000 7f9d5232d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:01.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:01.362+0000 7f9d5232d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:01.387 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:01Z"} 2024-03-04T17:33:01.388 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:01Z"} 2024-03-04T17:33:01.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:01.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:01.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:01.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:33:06.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:06.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:06.391 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:06.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:06.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:06.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:06.492+0000 7ff7aaf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:06.494 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:06.493+0000 7ff7aaf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:06.516 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:06Z"} 2024-03-04T17:33:06.535 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:06Z"} 2024-03-04T17:33:06.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:06.535 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:06.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:06.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.088500_457.1 2024-03-04T17:33:11.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:11.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:11.538 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:11.586 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:11.586 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:11.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:11.644+0000 7fbd22933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:11.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:11.645+0000 7fbd22933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:11.669 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:11Z"} 2024-03-04T17:33:11.669 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:11Z"} 2024-03-04T17:33:11.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:11.669 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:11.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:11.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:33:16.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:16.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:16.671 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:16.725 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:16.725 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:16.785 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:16.783+0000 7fe79362e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:16.785 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:16.784+0000 7fe79362e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:16.814 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:16Z"} 2024-03-04T17:33:16.815 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:16Z"} 2024-03-04T17:33:16.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:16.815 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:16.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:16.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T17:33:21.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:21.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:21.817 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:21.864 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:21.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:21.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:21.917+0000 7f9589f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:21.920 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:21.918+0000 7f9589f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:21.944 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:21Z"} 2024-03-04T17:33:21.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:21Z"} 2024-03-04T17:33:21.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:21.945 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:21.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:21.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T17:33:26.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:26.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:26.947 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:26.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:26.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:27.068 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:27.066+0000 7fdbe892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:27.068 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:27.067+0000 7fdbe892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:27.100 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:27Z"} 2024-03-04T17:33:27.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:27Z"} 2024-03-04T17:33:27.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:27.100 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:27.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:27.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.160094_460.1 2024-03-04T17:33:32.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:32.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:32.102 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:32.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:32.213+0000 7fd7eb62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:32.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:32.214+0000 7fd7eb62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:32.239 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:32Z"} 2024-03-04T17:33:32.239 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:32Z"} 2024-03-04T17:33:32.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:32.240 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:32.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:32.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:33:37.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:37.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:37.242 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:37.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:37.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:37.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:37.344+0000 7f6bd412d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:37.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:37.345+0000 7f6bd412d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:37.375 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:37Z"} 2024-03-04T17:33:37.375 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:37Z"} 2024-03-04T17:33:37.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:37.376 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:37.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:37.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:33:42.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:42.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:42.378 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:42.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:42.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:42.476+0000 7f8c3152f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:42.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:42.477+0000 7f8c3152f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:42.504 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:42Z"} 2024-03-04T17:33:42.504 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:42Z"} 2024-03-04T17:33:42.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:42.505 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:42.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:42.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:33:47.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:47.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:47.507 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:47.556 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:47.556 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:47.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:47.603+0000 7fa1a2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:47.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:47.604+0000 7fa1a2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:47.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:47Z"} 2024-03-04T17:33:47.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:47Z"} 2024-03-04T17:33:47.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:47.630 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:47.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:47.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T17:33:52.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:52.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:52.632 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:52.685 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:52.685 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:52.762+0000 7fc3c2711800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:52.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:52.763+0000 7fc3c2711800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:52.788 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:52Z"} 2024-03-04T17:33:52.789 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:52Z"} 2024-03-04T17:33:52.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:52.789 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:52.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:52.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:33:57.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:57.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:33:57.791 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:33:57.844 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:33:57.844 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:33:57.914 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:57.912+0000 7f643d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:57.914 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:33:57.913+0000 7f643d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:33:57.938 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:57Z"} 2024-03-04T17:33:57.939 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:57Z"} 2024-03-04T17:33:57.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:33:57.939 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:57.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:33:57.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T17:34:02.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:02.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:02.941 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:02.989 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:02.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:03.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:03.058+0000 7fa22db1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:03.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:03.059+0000 7fa22db1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:03.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:03Z"} 2024-03-04T17:34:03.094 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:03Z"} 2024-03-04T17:34:03.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:03.094 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:03.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:03.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709571357.881246_635.1 2024-03-04T17:34:08.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:08.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:08.095 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:08.148 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:08.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:08.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:08.298+0000 7f580a32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:08.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:08.301+0000 7f580a32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:08.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:08Z"} 2024-03-04T17:34:08.387 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:08Z"} 2024-03-04T17:34:08.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:08.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:08.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:08.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:34:13.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:13.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:13.388 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:13.467 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:13.467 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:13.576 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:13.574+0000 7f249568c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:13.576 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:13.575+0000 7f249568c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:13.608 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:13Z"} 2024-03-04T17:34:13.608 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:13Z"} 2024-03-04T17:34:13.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:13.608 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:13.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:13.609 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:34:18.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:18.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:18.610 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:18.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:18.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:18.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:18.710+0000 7f242313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:18.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:18.711+0000 7f242313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:18.739 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:18Z"} 2024-03-04T17:34:18.740 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:18Z"} 2024-03-04T17:34:18.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:18.740 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:18.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:18.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.160094_460.1 2024-03-04T17:34:23.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:23.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:23.742 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:23.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:23.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:23.848+0000 7f06dc33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:23.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:23.849+0000 7f06dc33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:23.879 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:23Z"} 2024-03-04T17:34:23.879 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:23Z"} 2024-03-04T17:34:23.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:23.880 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:23.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:23.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T17:34:28.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:28.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:28.882 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:28.932 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:28.932 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:28.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:28.994+0000 7fc528812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:28.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:28.995+0000 7fc528812800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:29.021 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:28Z"} 2024-03-04T17:34:29.021 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:28Z"} 2024-03-04T17:34:29.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:29.022 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:29.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:29.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T17:34:34.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:34.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:34.024 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:34.070 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:34.070 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:34.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:34.118+0000 7fa8fab0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:34.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:34.119+0000 7fa8fab0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:34.147 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:34Z"} 2024-03-04T17:34:34.148 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:34Z"} 2024-03-04T17:34:34.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:34.148 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:34.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:34.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:34:39.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:39.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:39.150 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:39.196 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:39.196 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:39.261 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:39.259+0000 7f0c14430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:39.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:39.260+0000 7f0c14430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:39.291 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:39Z"} 2024-03-04T17:34:39.291 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:39Z"} 2024-03-04T17:34:39.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:39.291 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:39.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:39.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:34:44.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:44.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:44.293 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:44.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:44.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:44.407 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:44.405+0000 7ff5f8731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:44.408 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:44.406+0000 7ff5f8731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:44.436 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:44Z"} 2024-03-04T17:34:44.437 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:44Z"} 2024-03-04T17:34:44.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:44.437 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:44.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:44.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570371.071168_455.1 2024-03-04T17:34:49.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:49.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:49.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:49.486 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:49.486 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:49.537 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:49.535+0000 7fd92368c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:49.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:49.536+0000 7fd92368c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:49.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:49Z"} 2024-03-04T17:34:49.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:49Z"} 2024-03-04T17:34:49.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:49.568 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:49.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:49.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T17:34:54.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:54.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:54.570 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:54.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:54.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:54.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:54.687+0000 7fdd2040f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:54.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:54.688+0000 7fdd2040f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:54.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:54Z"} 2024-03-04T17:34:54.721 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:54Z"} 2024-03-04T17:34:54.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:54.721 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:54.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:54.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T17:34:59.067 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:34:59.065+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:34:59.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:59.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:34:59.723 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:34:59.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:34:59.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:34:59.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:59.842+0000 7f7400024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:59.844 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:34:59.843+0000 7f7400024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:34:59.880 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:59Z"} 2024-03-04T17:34:59.880 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:59Z"} 2024-03-04T17:34:59.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:34:59.881 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:59.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:34:59.881 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:35:04.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:04.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:04.883 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:04.933 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:04.933 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:04.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:04.985+0000 7fc970b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:04.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:04.986+0000 7fc970b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:05.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:04Z"} 2024-03-04T17:35:05.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:04Z"} 2024-03-04T17:35:05.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:05.011 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:05.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:05.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:35:10.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:10.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:10.014 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:10.062 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:10.062 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:10.113 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:10.111+0000 7f29d7b42800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:10.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:10.112+0000 7f29d7b42800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:10Z"} 2024-03-04T17:35:10.137 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:10Z"} 2024-03-04T17:35:10.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:10.138 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:10.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:10.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:35:15.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:15.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:15.139 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:15.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:15.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:15.249 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:15.247+0000 7f872d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:15.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:15.248+0000 7f872d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:15.273 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:15Z"} 2024-03-04T17:35:15.273 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:15Z"} 2024-03-04T17:35:15.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:15.273 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:15.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:15.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:35:20.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:20.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:20.275 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:20.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:20.385+0000 7fc8b728a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:20.386 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:20.385+0000 7fc8b728a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:20.412 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:20Z"} 2024-03-04T17:35:20.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:20Z"} 2024-03-04T17:35:20.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:20.413 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:20.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:20.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T17:35:25.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:25.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:25.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:25.467 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:25.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:25.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:25.527+0000 7fbb8d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:25.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:25.528+0000 7fbb8d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:25.557 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:25Z"} 2024-03-04T17:35:25.557 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:25Z"} 2024-03-04T17:35:25.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:25.557 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:25.557 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:25.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T17:35:30.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:30.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:30.559 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:30.609 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:30.609 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:30.671 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:30.669+0000 7fb94fc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:30.672 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:30.670+0000 7fb94fc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:30.701 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:30Z"} 2024-03-04T17:35:30.701 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:30Z"} 2024-03-04T17:35:30.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:30.702 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:30.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:30.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:35:35.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:35.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:35.704 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:35.786 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:35.786 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:35.916 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:35.914+0000 7fd25932f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:35.917 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:35.915+0000 7fd25932f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:35.948 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:35Z"} 2024-03-04T17:35:35.948 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:35Z"} 2024-03-04T17:35:35.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:35.949 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:35.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:35.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:35:40.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:40.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:40.950 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:41.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:41.146+0000 7f8deef1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:41.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:41.147+0000 7f8deef1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:41.176 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:41Z"} 2024-03-04T17:35:41.176 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:41Z"} 2024-03-04T17:35:41.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:41.176 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:41.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:41.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.155110_458.1 2024-03-04T17:35:46.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:46.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:46.178 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:46.224 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:46.225 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:46.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:46.271+0000 7fd1b5211800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:46.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:46.272+0000 7fd1b5211800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:46.302 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:46Z"} 2024-03-04T17:35:46.302 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:46Z"} 2024-03-04T17:35:46.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:46.303 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:46.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:46.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:35:51.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:51.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:51.305 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:51.352 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:51.352 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:51.410 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:51.408+0000 7f9b10224800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:51.410 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:51.409+0000 7f9b10224800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:51.435 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:51Z"} 2024-03-04T17:35:51.436 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:51Z"} 2024-03-04T17:35:51.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:51.436 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:51.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:51.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:35:56.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:56.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:35:56.438 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:35:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:35:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:35:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:56.535+0000 7f815002e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:56.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:35:56.536+0000 7f815002e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:35:56.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:56Z"} 2024-03-04T17:35:56.561 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:56Z"} 2024-03-04T17:35:56.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:35:56.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:56.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:35:56.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T17:36:01.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:01.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:01.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:01.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:01.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:01.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:01.680+0000 7f411e02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:01.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:01.681+0000 7f411e02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:01.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:01Z"} 2024-03-04T17:36:01.707 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:01Z"} 2024-03-04T17:36:01.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:01.707 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:01.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:01.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:36:06.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:06.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:06.709 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:06.759 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:06.759 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:06.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:06.811+0000 7f1faff2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:06.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:06.812+0000 7f1faff2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:06.841 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:06Z"} 2024-03-04T17:36:06.842 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:06Z"} 2024-03-04T17:36:06.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:06.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:06.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:06.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T17:36:11.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:11.844 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:11.955 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:11.953+0000 7ffb54b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:11.954+0000 7ffb54b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:11.988 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T17:36:11.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:11.988 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:11.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:11.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:36:16.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:16.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:17.042 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:17.042 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:17.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:17.094+0000 7f83dcb33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:17.095+0000 7f83dcb33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:17.118 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.119 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T17:36:17.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:17.119 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:17.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:17.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.192640_463.1 2024-03-04T17:36:22.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:22.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:22.124 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:22.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:22.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:22.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:22.235+0000 7f6b6c71f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:22.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:22.236+0000 7f6b6c71f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:22.268 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.167376_461.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.167376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:22Z"} 2024-03-04T17:36:22.268 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.167376_461.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.167376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:22Z"} 2024-03-04T17:36:22.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.167376_461.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.167376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:22.268 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:22.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.167376_461.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:22.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.167376_461.1 2024-03-04T17:36:27.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:27.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:27.270 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:27.323 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:27.324 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:27.388+0000 7f8092c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:27.389+0000 7f8092c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:27.414 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:27Z"} 2024-03-04T17:36:27.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:27Z"} 2024-03-04T17:36:27.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:27.415 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:27.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:27.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570988.644921_566.1 2024-03-04T17:36:32.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:32.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:32.416 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:32.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:32.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:32.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:32.558+0000 7f60a5f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:32.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:32.560+0000 7f60a5f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:32.662 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:32Z"} 2024-03-04T17:36:32.662 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:32Z"} 2024-03-04T17:36:32.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:32.663 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:32.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:32.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T17:36:37.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:37.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:37.665 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:37.714 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:37.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:37.884+0000 7faa6092e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:37.891 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:37.889+0000 7faa6092e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:37.918 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:37Z"} 2024-03-04T17:36:37.918 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:37Z"} 2024-03-04T17:36:37.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:37.919 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:37.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:37.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T17:36:42.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:42.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:42.920 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:42.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:42.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:43.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:43.112+0000 7f0752e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:43.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:43.113+0000 7f0752e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:43Z"} 2024-03-04T17:36:43.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:43Z"} 2024-03-04T17:36:43.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:43.143 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:43.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:43.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T17:36:48.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:48.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:48.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:48.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:48.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:48.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:48.258+0000 7fc43e911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:48.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:48.259+0000 7fc43e911800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:48.289 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:48Z"} 2024-03-04T17:36:48.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:48Z"} 2024-03-04T17:36:48.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.218063_467.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.218063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:48.290 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:48.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.218063_467.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:48.290 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.218063_467.1 2024-03-04T17:36:53.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:53.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:53.292 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:53.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:53.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:53.395+0000 7fbf47933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:53.396+0000 7fbf47933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:53.425 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:53Z"} 2024-03-04T17:36:53.425 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:53Z"} 2024-03-04T17:36:53.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:53.426 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:53.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:53.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T17:36:58.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:58.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:36:58.428 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:36:58.474 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:36:58.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:36:58.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:58.524+0000 7f77f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:58.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:36:58.525+0000 7f77f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:36:58.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:58Z"} 2024-03-04T17:36:58.550 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:58Z"} 2024-03-04T17:36:58.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:36:58.550 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:58.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:36:58.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.015683_453.1 2024-03-04T17:37:03.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:03.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:03.552 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:03.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:03.598 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:03.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:03.661+0000 7ffa7892d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:03.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:03.661+0000 7ffa7892d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:03.692 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:03Z"} 2024-03-04T17:37:03.692 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:03Z"} 2024-03-04T17:37:03.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:03.692 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:03.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:03.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T17:37:08.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:08.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:08.694 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:08.741 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:08.741 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:08.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:08.794+0000 7f54d0044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:08.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:08.795+0000 7f54d0044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:08.819 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:08Z"} 2024-03-04T17:37:08.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:08Z"} 2024-03-04T17:37:08.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:08.820 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:08.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:08.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T17:37:13.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:13.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:13.822 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:13.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:13.868 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:13.933 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:13.932+0000 7ff51eb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:13.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:13.933+0000 7ff51eb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:13.957 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:13Z"} 2024-03-04T17:37:13.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:13Z"} 2024-03-04T17:37:13.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:13.958 DEBUG:rgw_multi.tests:log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:13.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:13.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570371.071168_455.1 2024-03-04T17:37:18.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-03-04T17:37:18.962 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=sbnvju-69 2024-03-04T17:37:18.962 DEBUG:boto:path=/sbnvju-69/ 2024-03-04T17:37:18.962 DEBUG:boto:auth_path=/sbnvju-69/ 2024-03-04T17:37:18.962 DEBUG:boto:Method: PUT 2024-03-04T17:37:18.962 DEBUG:boto:Path: /sbnvju-69/ 2024-03-04T17:37:18.962 DEBUG:boto:Data: 2024-03-04T17:37:18.963 DEBUG:boto:Headers: {} 2024-03-04T17:37:18.963 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:18.963 DEBUG:boto:Port: 8000 2024-03-04T17:37:18.963 DEBUG:boto:Params: {} 2024-03-04T17:37:18.963 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:37:18.963 DEBUG:boto:Token: None 2024-03-04T17:37:18.964 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:37:18 GMT /sbnvju-69/ 2024-03-04T17:37:18.964 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:uTmeewIU9eu7zNZOaiORUicsU2w= 2024-03-04T17:37:18.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uTmeewIU9eu7zNZOaiORUicsU2w=', 'Content-Length': '0'} 2024-03-04T17:37:18.988 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000776ecfb23a5223d9-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:18.988 DEBUG:boto:path=/sbnvju-69/ 2024-03-04T17:37:18.988 DEBUG:boto:auth_path=/sbnvju-69/ 2024-03-04T17:37:18.988 DEBUG:boto:Method: HEAD 2024-03-04T17:37:18.989 DEBUG:boto:Path: /sbnvju-69/ 2024-03-04T17:37:18.989 DEBUG:boto:Data: 2024-03-04T17:37:18.989 DEBUG:boto:Headers: {} 2024-03-04T17:37:18.989 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:18.989 DEBUG:boto:Port: 8000 2024-03-04T17:37:18.989 DEBUG:boto:Params: {} 2024-03-04T17:37:18.989 DEBUG:boto:Token: None 2024-03-04T17:37:18.989 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:37:18 GMT /sbnvju-69/ 2024-03-04T17:37:18.989 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:0gOQRVpjKnamawra3k9w8TJ5WL4= 2024-03-04T17:37:18.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0gOQRVpjKnamawra3k9w8TJ5WL4=', 'Content-Length': '0'} 2024-03-04T17:37:18.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2ab08452aaf7dfe-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:18.992 DEBUG:boto:path=/sbnvju-69/a 2024-03-04T17:37:18.992 DEBUG:boto:auth_path=/sbnvju-69/a 2024-03-04T17:37:18.993 DEBUG:boto:Method: PUT 2024-03-04T17:37:18.993 DEBUG:boto:Path: /sbnvju-69/a 2024-03-04T17:37:18.993 DEBUG:boto:Data: 2024-03-04T17:37:18.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:37:18.993 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:18.993 DEBUG:boto:Port: 8000 2024-03-04T17:37:18.993 DEBUG:boto:Params: {} 2024-03-04T17:37:18.993 DEBUG:boto:Token: None 2024-03-04T17:37:18.993 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:37:18 GMT /sbnvju-69/a 2024-03-04T17:37:18.993 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:j00LshalDHcTfKFgRSLR9JkPKzY= 2024-03-04T17:37:18.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j00LshalDHcTfKFgRSLR9JkPKzY='} 2024-03-04T17:37:19.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000860e2c2f00b727a2-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.001 DEBUG:boto:path=/sbnvju-69/ 2024-03-04T17:37:19.001 DEBUG:boto:auth_path=/sbnvju-69/ 2024-03-04T17:37:19.001 DEBUG:boto:Method: HEAD 2024-03-04T17:37:19.001 DEBUG:boto:Path: /sbnvju-69/ 2024-03-04T17:37:19.001 DEBUG:boto:Data: 2024-03-04T17:37:19.001 DEBUG:boto:Headers: {} 2024-03-04T17:37:19.001 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.001 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.001 DEBUG:boto:Params: {} 2024-03-04T17:37:19.002 DEBUG:boto:Token: None 2024-03-04T17:37:19.002 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/ 2024-03-04T17:37:19.002 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T17:37:19.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T17:37:19.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cffe6e7e521a11e7-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.004 DEBUG:boto:path=/sbnvju-69/b 2024-03-04T17:37:19.004 DEBUG:boto:auth_path=/sbnvju-69/b 2024-03-04T17:37:19.005 DEBUG:boto:Method: PUT 2024-03-04T17:37:19.005 DEBUG:boto:Path: /sbnvju-69/b 2024-03-04T17:37:19.005 DEBUG:boto:Data: 2024-03-04T17:37:19.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:37:19.005 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.005 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.005 DEBUG:boto:Params: {} 2024-03-04T17:37:19.005 DEBUG:boto:Token: None 2024-03-04T17:37:19.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/b 2024-03-04T17:37:19.005 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:wz/sYXk1+f4+yFnxV13Rus6cga8= 2024-03-04T17:37:19.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wz/sYXk1+f4+yFnxV13Rus6cga8='} 2024-03-04T17:37:19.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086901d2c24984849-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.048 DEBUG:boto:path=/sbnvju-69/ 2024-03-04T17:37:19.049 DEBUG:boto:auth_path=/sbnvju-69/ 2024-03-04T17:37:19.049 DEBUG:boto:Method: HEAD 2024-03-04T17:37:19.049 DEBUG:boto:Path: /sbnvju-69/ 2024-03-04T17:37:19.049 DEBUG:boto:Data: 2024-03-04T17:37:19.049 DEBUG:boto:Headers: {} 2024-03-04T17:37:19.049 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.049 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.049 DEBUG:boto:Params: {} 2024-03-04T17:37:19.049 DEBUG:boto:Token: None 2024-03-04T17:37:19.049 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/ 2024-03-04T17:37:19.049 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T17:37:19.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T17:37:19.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7e84c518843604-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.052 DEBUG:boto:path=/sbnvju-69/c 2024-03-04T17:37:19.053 DEBUG:boto:auth_path=/sbnvju-69/c 2024-03-04T17:37:19.053 DEBUG:boto:Method: PUT 2024-03-04T17:37:19.053 DEBUG:boto:Path: /sbnvju-69/c 2024-03-04T17:37:19.053 DEBUG:boto:Data: 2024-03-04T17:37:19.053 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:37:19.053 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.053 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.053 DEBUG:boto:Params: {} 2024-03-04T17:37:19.053 DEBUG:boto:Token: None 2024-03-04T17:37:19.053 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/c 2024-03-04T17:37:19.053 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:y7nteBr2P47ZX6g5ktSv4YOv8qo= 2024-03-04T17:37:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y7nteBr2P47ZX6g5ktSv4YOv8qo='} 2024-03-04T17:37:19.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e74c6f357904455a-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.060 DEBUG:boto:path=/sbnvju-69/ 2024-03-04T17:37:19.060 DEBUG:boto:auth_path=/sbnvju-69/ 2024-03-04T17:37:19.060 DEBUG:boto:Method: HEAD 2024-03-04T17:37:19.060 DEBUG:boto:Path: /sbnvju-69/ 2024-03-04T17:37:19.061 DEBUG:boto:Data: 2024-03-04T17:37:19.061 DEBUG:boto:Headers: {} 2024-03-04T17:37:19.061 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.061 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.061 DEBUG:boto:Params: {} 2024-03-04T17:37:19.061 DEBUG:boto:Token: None 2024-03-04T17:37:19.061 DEBUG:boto:StringToSign: HEAD Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/ 2024-03-04T17:37:19.061 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T17:37:19.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T17:37:19.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa96d85e00556157-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.064 DEBUG:boto:path=/sbnvju-69/d 2024-03-04T17:37:19.064 DEBUG:boto:auth_path=/sbnvju-69/d 2024-03-04T17:37:19.064 DEBUG:boto:Method: PUT 2024-03-04T17:37:19.064 DEBUG:boto:Path: /sbnvju-69/d 2024-03-04T17:37:19.065 DEBUG:boto:Data: 2024-03-04T17:37:19.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T17:37:19.065 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:37:19.065 DEBUG:boto:Port: 8000 2024-03-04T17:37:19.065 DEBUG:boto:Params: {} 2024-03-04T17:37:19.065 DEBUG:boto:Token: None 2024-03-04T17:37:19.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 04 Mar 2024 17:37:19 GMT /sbnvju-69/d 2024-03-04T17:37:19.065 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:vSzyF2LKNhiCKH+hHR3eCCbENm0= 2024-03-04T17:37:19.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vSzyF2LKNhiCKH+hHR3eCCbENm0='} 2024-03-04T17:37:19.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd3eaa3545396345-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:37:19.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:19.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:19.109 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:19.158 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:19.158 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:19.209 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:37:19.233 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709573838.984527_954.1","last_update":"2024-03-04T17:37:18.984527Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:37:19.233 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:19.233 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:37:19.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:19.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:19.234 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:19.311 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:19.311 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:19.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:19.401+0000 7f7250d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:19.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:19.402+0000 7f7250d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:19.430 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:19Z"} 2024-03-04T17:37:19.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:19Z"} 2024-03-04T17:37:19.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:19.430 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:19.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:19.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.975603_447.1 2024-03-04T17:37:24.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:24.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:24.431 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:24.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:24.558+0000 7f1724d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:24.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:24.558+0000 7f1724d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:24.591 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:37:24.589+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:37:24.592 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:24Z"} 2024-03-04T17:37:24.592 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:24Z"} 2024-03-04T17:37:24.592 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:24.593 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:24.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:24.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570371.064903_454.1 2024-03-04T17:37:29.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:29.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:29.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:29.704 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:29.702+0000 7f0002f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:29.705 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:29.703+0000 7f0002f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:29Z"} 2024-03-04T17:37:29.728 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:29Z"} 2024-03-04T17:37:29.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:29.728 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:29.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:29.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.692350_570.1 2024-03-04T17:37:34.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:34.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:34.729 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:34.780 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:34.780 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:34.836 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:34.834+0000 7f430bf4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:34.836 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:34.834+0000 7f430bf4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:34.860 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:34Z"} 2024-03-04T17:37:34.861 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:34Z"} 2024-03-04T17:37:34.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:34.861 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:34.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:34.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T17:37:39.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:39.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:39.863 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:39.909 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:39.909 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:39.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:39.972+0000 7fb496412800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:39.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:39.973+0000 7fb496412800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:39.998 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:39Z"} 2024-03-04T17:37:39.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:39Z"} 2024-03-04T17:37:39.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:39.999 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:39.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:39.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T17:37:45.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:45.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:45.001 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:45.047 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:45.047 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:45.133 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:45.131+0000 7fdbf9b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:45.134 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:45.132+0000 7fdbf9b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:45.218 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:45Z"} 2024-03-04T17:37:45.218 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:45Z"} 2024-03-04T17:37:45.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:45.218 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:45.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:45.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.088500_457.1 2024-03-04T17:37:50.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:50.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:50.220 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:50.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:50.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:50.344+0000 7f9572430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:50.344+0000 7f9572430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:50.370 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:50Z"} 2024-03-04T17:37:50.370 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:50Z"} 2024-03-04T17:37:50.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:50.371 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:50.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:50.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.084490_456.1 2024-03-04T17:37:55.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:55.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:37:55.372 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:37:55.424 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:37:55.424 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:37:55.494 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:55.492+0000 7f415d6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:55.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:37:55.493+0000 7f415d6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:37:55.521 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:55Z"} 2024-03-04T17:37:55.521 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:55Z"} 2024-03-04T17:37:55.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:37:55.521 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:55.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:37:55.522 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570370.000296_451.1 2024-03-04T17:38:00.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:00.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:00.523 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:00.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:00.571 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:00.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:00.621+0000 7fe5d9a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:00.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:00.622+0000 7fe5d9a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:00.646 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:00Z"} 2024-03-04T17:38:00.646 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:00Z"} 2024-03-04T17:38:00.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:00.647 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:00.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:00.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.972318_445.1 2024-03-04T17:38:05.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:05.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:05.649 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:05.695 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:05.695 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:05.755 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:05.753+0000 7f8cfc611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:05.756 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:05.754+0000 7f8cfc611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:05.781 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:05Z"} 2024-03-04T17:38:05.781 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:05Z"} 2024-03-04T17:38:05.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:05.781 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:05.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:05.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.158768_459.1 2024-03-04T17:38:10.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:10.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:10.783 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:10.829 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:10.829 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:10.882 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:10.881+0000 7fce33d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:10.883 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:10.882+0000 7fce33d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:10.906 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:10Z"} 2024-03-04T17:38:10.906 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:10Z"} 2024-03-04T17:38:10.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:10.906 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:10.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:10.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T17:38:15.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:15.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:15.908 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:15.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:15.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:16.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:16.007+0000 7fdd28d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:16.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:16.008+0000 7fdd28d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:16.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:16Z"} 2024-03-04T17:38:16.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:16Z"} 2024-03-04T17:38:16.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:16.033 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:16.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:16.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.189913_462.1 2024-03-04T17:38:21.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:21.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:21.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:21.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:21.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:21.146 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:21.144+0000 7f1469e67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:21.147 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:21.145+0000 7f1469e67800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:21.172 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.221520_469.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.221520Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:21Z"} 2024-03-04T17:38:21.172 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.221520_469.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.221520Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:21Z"} 2024-03-04T17:38:21.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.221520_469.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.221520Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:21.173 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:21.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:21.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.221520_469.1 2024-03-04T17:38:26.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:26.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:26.175 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:26.223 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:26.223 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:26.278+0000 7f5c0574b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:26.279+0000 7f5c0574b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:26.307 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:26Z"} 2024-03-04T17:38:26.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:26Z"} 2024-03-04T17:38:26.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:26.308 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:26.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:26.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.972318_445.1 2024-03-04T17:38:31.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:31.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:31.310 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:31.359 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:31.359 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:31.420 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:31.418+0000 7f76c8844800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:31.420 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:31.419+0000 7f76c8844800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:31.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:31Z"} 2024-03-04T17:38:31.443 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:31Z"} 2024-03-04T17:38:31.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:31.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:31.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:31.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.215333_466.1 2024-03-04T17:38:36.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:36.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:36.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:36.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:36.559+0000 7f2494f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:36.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:36.560+0000 7f2494f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:36.591 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:36Z"} 2024-03-04T17:38:36.591 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:36Z"} 2024-03-04T17:38:36.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:36.592 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:36.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:36.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.215333_466.1 2024-03-04T17:38:41.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:41.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:41.593 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:41.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:41.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:41.705+0000 7ff50622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:41.706+0000 7ff50622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:41.733 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:41Z"} 2024-03-04T17:38:41.734 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:41Z"} 2024-03-04T17:38:41.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:41.734 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:41.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:41.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T17:38:46.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:46.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:46.735 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:46.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:46.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:46.854 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:46.852+0000 7f07d1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:46.855 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:46.853+0000 7f07d1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:46.882 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:46Z"} 2024-03-04T17:38:46.882 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:46Z"} 2024-03-04T17:38:46.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:46.882 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:46.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:46.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.975603_447.1 2024-03-04T17:38:51.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:51.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:51.884 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:51.935 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:51.935 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:52.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:52.003+0000 7fe2bbc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:52.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:52.004+0000 7fe2bbc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:52.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:52Z"} 2024-03-04T17:38:52.031 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:52Z"} 2024-03-04T17:38:52.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:52.032 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:52.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:52.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.158768_459.1 2024-03-04T17:38:57.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:57.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:38:57.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:38:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:38:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:38:57.146 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:57.144+0000 7fc849333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:57.147 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:38:57.145+0000 7fc849333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:38:57.169 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:57Z"} 2024-03-04T17:38:57.169 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:57Z"} 2024-03-04T17:38:57.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:38:57.170 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:57.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:38:57.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T17:39:02.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:02.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:02.171 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:02.224 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:02.224 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:02.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:02.286+0000 7f96fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:02.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:02.286+0000 7f96fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:02.315 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:02Z"} 2024-03-04T17:39:02.315 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:02Z"} 2024-03-04T17:39:02.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:02.316 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:02.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:02.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571357.885240_641.1 2024-03-04T17:39:07.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:07.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:07.317 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:07.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:07.413+0000 7fec7fe22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:07.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:07.413+0000 7fec7fe22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:07.444 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:07Z"} 2024-03-04T17:39:07.444 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:07Z"} 2024-03-04T17:39:07.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:07.445 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:07.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:07.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.612016_560.1 2024-03-04T17:39:12.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:12.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:12.447 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:12.494 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:12.494 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:12.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:12.564+0000 7f3ecdf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:12.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:12.565+0000 7f3ecdf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:12.590 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:12Z"} 2024-03-04T17:39:12.590 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:12Z"} 2024-03-04T17:39:12.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:12.591 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:12.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:12.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.116657_707.1 2024-03-04T17:39:17.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:17.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:17.592 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:17.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:17.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:17.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:17.701+0000 7f3565e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:17.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:17.702+0000 7f3565e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:17.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:17Z"} 2024-03-04T17:39:17.727 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:17Z"} 2024-03-04T17:39:17.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:17.728 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:17.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:17.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T17:39:22.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:22.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:22.729 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:22.785 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:22.785 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:22.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:22.838+0000 7f3af6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:22.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:22.839+0000 7f3af6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:22.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:22Z"} 2024-03-04T17:39:22.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:22Z"} 2024-03-04T17:39:22.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:22.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:22.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:22.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.987908_703.1 2024-03-04T17:39:27.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:27.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:27.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:27.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:27.922 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:27.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:27.982+0000 7f878cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:27.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:27.982+0000 7f878cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:28.012 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:27Z"} 2024-03-04T17:39:28.013 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:27Z"} 2024-03-04T17:39:28.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:28.013 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:28.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:28.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T17:39:33.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:33.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:33.014 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:33.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:33.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:33.132 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:33.131+0000 7f2b7273c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:33.133 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:33.132+0000 7f2b7273c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:33.157 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:33Z"} 2024-03-04T17:39:33.157 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:33Z"} 2024-03-04T17:39:33.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:33.157 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:33.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:33.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T17:39:38.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:38.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:38.159 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:38.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:38.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:38.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:38.295+0000 7f0033d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:38.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:38.297+0000 7f0033d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:38.395 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:38Z"} 2024-03-04T17:39:38.395 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:38Z"} 2024-03-04T17:39:38.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:38.396 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:38.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:38.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.242177_729.1 2024-03-04T17:39:43.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:43.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:43.397 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:43.483 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:43.483 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:43.595+0000 7f3cea212800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:43.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:43.596+0000 7f3cea212800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:43.622 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:43Z"} 2024-03-04T17:39:43.622 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:43Z"} 2024-03-04T17:39:43.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:43.622 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:43.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:43.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T17:39:48.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:48.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:48.624 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:48.671 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:48.671 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:48.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:48.720+0000 7f760eb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:48.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:48.721+0000 7f760eb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:48.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:48Z"} 2024-03-04T17:39:48.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:48Z"} 2024-03-04T17:39:48.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:48.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:48.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:48.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T17:39:53.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:53.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:53.749 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:53.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:53.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:53.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:53.849+0000 7fa332016800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:53.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:53.850+0000 7fa332016800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:53.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:53Z"} 2024-03-04T17:39:53.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:53Z"} 2024-03-04T17:39:53.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:53.875 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:53.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:53.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T17:39:58.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:58.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:39:58.876 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:39:58.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:39:58.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:39:58.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:58.982+0000 7f638b62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:58.985 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:39:58.983+0000 7f638b62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:39:59.008 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:58Z"} 2024-03-04T17:39:59.008 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:58Z"} 2024-03-04T17:39:59.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:39:59.009 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:59.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:39:59.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.613615_561.1 2024-03-04T17:40:01.345 INFO:tasks.rgw.c1.client.0.smithi027.stdout:2024-03-04T17:40:01.343+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T17:40:04.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:04.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:04.010 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:04.056 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:04.056 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:04.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:04.118+0000 7f5eb5731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:04.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:04.119+0000 7f5eb5731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:04.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:04Z"} 2024-03-04T17:40:04.145 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:04Z"} 2024-03-04T17:40:04.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:04.145 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:04.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:04.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.616944_562.1 2024-03-04T17:40:09.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:09.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:09.147 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:09.195 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:09.195 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:09.255 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:09.254+0000 7f6ed3444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:09.256 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:09.254+0000 7f6ed3444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:09.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:09Z"} 2024-03-04T17:40:09.283 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:09Z"} 2024-03-04T17:40:09.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:09.283 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:09.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:09.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.641846_565.1 2024-03-04T17:40:14.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:14.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:14.285 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:14.331 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:14.331 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:14.383+0000 7fa0d7c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:14.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:14.384+0000 7fa0d7c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:14.417 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:14Z"} 2024-03-04T17:40:14.418 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:14Z"} 2024-03-04T17:40:14.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:14.418 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:14.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:14.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T17:40:19.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:19.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:19.420 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:19.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:19.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:19.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:19.527+0000 7f36bb636800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:19.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:19.528+0000 7f36bb636800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:19.553 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:19Z"} 2024-03-04T17:40:19.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:19Z"} 2024-03-04T17:40:19.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:19.554 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:19.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:19.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T17:40:24.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:24.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:24.556 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:24.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:24.608 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:24.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:24.660+0000 7ff1a47af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:24.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:24.661+0000 7ff1a47af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:24.685 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:24Z"} 2024-03-04T17:40:24.686 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:24Z"} 2024-03-04T17:40:24.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:24.686 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:24.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:24.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.613615_561.1 2024-03-04T17:40:29.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:29.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:29.689 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:29.753 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:29.753 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:29.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:29.925+0000 7f636bf24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:29.927 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:29.926+0000 7f636bf24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:29.957 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:29Z"} 2024-03-04T17:40:29.957 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:29Z"} 2024-03-04T17:40:29.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:29.957 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:29.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:29.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571357.881246_635.1 2024-03-04T17:40:34.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:34.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:34.959 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:35.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:35.036 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:35.125 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:35.124+0000 7fe891122800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:35.126 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:35.125+0000 7fe891122800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:35.152 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:35Z"} 2024-03-04T17:40:35.152 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:35Z"} 2024-03-04T17:40:35.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:35.152 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:35.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:35.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.002120_706.1 2024-03-04T17:40:40.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:40.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:40.154 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:40.227 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:40.227 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:40.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:40.301+0000 7fe957b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:40.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:40.303+0000 7fe957b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:40.344 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:40Z"} 2024-03-04T17:40:40.344 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:40Z"} 2024-03-04T17:40:40.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:40.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:40.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:40.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T17:40:45.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:45.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:45.346 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:45.422 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:45.422 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:45.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:45.566+0000 7f028dac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:45.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:45.567+0000 7f028dac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:45.593 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:45Z"} 2024-03-04T17:40:45.594 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:45Z"} 2024-03-04T17:40:45.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:45.594 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:45.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:45.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T17:40:50.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:50.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:50.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:50.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:50.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:50.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:50.786+0000 7f703c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:50.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:50.787+0000 7f703c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:50.822 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:50Z"} 2024-03-04T17:40:50.823 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:50Z"} 2024-03-04T17:40:50.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:50.823 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:50.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:50.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.116657_707.1 2024-03-04T17:40:55.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:55.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:40:55.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:40:55.899 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:40:55.899 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:40:55.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:55.987+0000 7f240d32b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:55.989 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:40:55.988+0000 7f240d32b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:40:56.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:55Z"} 2024-03-04T17:40:56.012 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:55Z"} 2024-03-04T17:40:56.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:40:56.012 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:56.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:40:56.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T17:41:01.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:01.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:01.014 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:01.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:01.073 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:01.134 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:01.133+0000 7fb55f89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:01.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:01.134+0000 7fb55f89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:01.159 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:01Z"} 2024-03-04T17:41:01.159 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:01Z"} 2024-03-04T17:41:01.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:01.159 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:01.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:01.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T17:41:06.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:06.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:06.161 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:06.277+0000 7f00bc52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:06.277+0000 7f00bc52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:06.306 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:06Z"} 2024-03-04T17:41:06.306 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:06Z"} 2024-03-04T17:41:06.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:06.306 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:06.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:06.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.987908_703.1 2024-03-04T17:41:11.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:11.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:11.308 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:11.354 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:11.354 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:11.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:11.410+0000 7fc13f933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:11.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:11.411+0000 7fc13f933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:11.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:11Z"} 2024-03-04T17:41:11.442 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:11Z"} 2024-03-04T17:41:11.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:11.442 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:11.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:11.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571980.296697_734.1 2024-03-04T17:41:16.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:16.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:16.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:16.490 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:16.490 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:16.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:16.542+0000 7f2e9fc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:16.544 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:16.543+0000 7f2e9fc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:16.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:16Z"} 2024-03-04T17:41:16.567 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:16Z"} 2024-03-04T17:41:16.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:16.567 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:16.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:16.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.616944_562.1 2024-03-04T17:41:21.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:21.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:21.568 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:21.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:21.615 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:21.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:21.678+0000 7f992043a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:21.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:21.678+0000 7f992043a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:21.709 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:21Z"} 2024-03-04T17:41:21.710 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:21Z"} 2024-03-04T17:41:21.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:21.710 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:21.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:21.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.612016_560.1 2024-03-04T17:41:26.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:26.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:26.711 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:26.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:26.821+0000 7ff11342e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:26.823 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:26.822+0000 7ff11342e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:26.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:26Z"} 2024-03-04T17:41:26.847 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:26Z"} 2024-03-04T17:41:26.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:26.848 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:26.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:26.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T17:41:31.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:31.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:31.849 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:31.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:31.955+0000 7fdba360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:31.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:31.956+0000 7fdba360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:31.980 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:31Z"} 2024-03-04T17:41:31.980 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:31Z"} 2024-03-04T17:41:31.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:31.981 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:31.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:31.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.828757_695.1 2024-03-04T17:41:36.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:36.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:36.982 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:37.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:37.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:37.102 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:37.101+0000 7fd6fe20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:37.103 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:37.102+0000 7fd6fe20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:37.136 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:37Z"} 2024-03-04T17:41:37.136 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:37Z"} 2024-03-04T17:41:37.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:37.137 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:37.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:37.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T17:41:42.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:42.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:42.138 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:42.186 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:42.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:42.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:42.236+0000 7ff8d0c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:42.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:42.237+0000 7ff8d0c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:42.261 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.754989_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.754989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:42Z"} 2024-03-04T17:41:42.262 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.754989_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.754989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:42Z"} 2024-03-04T17:41:42.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572290.754989_772.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:11:30.754989Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:42.262 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:42.262 DEBUG:rgw_multi.tests:sync_status={0: '1_1709572290.754989_772.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:42.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709572290.754989_772.1 2024-03-04T17:41:47.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:47.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:47.264 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:47.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:47.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:47.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:47.364+0000 7fcb18730800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:47.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:47.365+0000 7fcb18730800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:47.389 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:47Z"} 2024-03-04T17:41:47.389 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:47Z"} 2024-03-04T17:41:47.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:47.389 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:47.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:47.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T17:41:52.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:52.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:52.391 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:52.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:52.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:52.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:52.508+0000 7f88a7a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:52.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:52.509+0000 7f88a7a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:52.533 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:52Z"} 2024-03-04T17:41:52.534 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:52Z"} 2024-03-04T17:41:52.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:52.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:52.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:52.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T17:41:57.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:57.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:41:57.535 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:41:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:41:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:41:57.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:57.637+0000 7f829582f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:57.639 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:41:57.638+0000 7f829582f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:41:57.664 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:57Z"} 2024-03-04T17:41:57.664 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:57Z"} 2024-03-04T17:41:57.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:41:57.665 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:57.665 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:41:57.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.828757_695.1 2024-03-04T17:42:02.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:02.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:02.667 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:02.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:02.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:02.762+0000 7f6ef6c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:02.763+0000 7f6ef6c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:02.787 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:02Z"} 2024-03-04T17:42:02.787 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:02Z"} 2024-03-04T17:42:02.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:02.787 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:02.787 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:02.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.692350_570.1 2024-03-04T17:42:07.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:07.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:07.789 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:07.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:07.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:07.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:07.909+0000 7f13d081e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:07.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:07.910+0000 7f13d081e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:07.935 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:07Z"} 2024-03-04T17:42:07.935 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:07Z"} 2024-03-04T17:42:07.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:07.935 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:07.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:07.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T17:42:12.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:12.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:12.937 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:12.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:12.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:13.045 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:13.044+0000 7f6a78444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:13.046 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:13.045+0000 7f6a78444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:13.069 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:13Z"} 2024-03-04T17:42:13.069 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:13Z"} 2024-03-04T17:42:13.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:13.069 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:13.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:13.069 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.641846_565.1 2024-03-04T17:42:18.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:18.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:18.071 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:18.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:18.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:18.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:18.200+0000 7f28fd511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:18.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:18.202+0000 7f28fd511800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:18.290 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:18Z"} 2024-03-04T17:42:18.291 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:18Z"} 2024-03-04T17:42:18.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:18.291 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:18.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:18.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T17:42:23.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:23.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:23.293 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:23.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:23.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:23.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:23.441+0000 7f5e33267800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:23.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:23.442+0000 7f5e33267800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:23.479 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:23Z"} 2024-03-04T17:42:23.480 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:23Z"} 2024-03-04T17:42:23.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:23.480 DEBUG:rgw_multi.tests:log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:23.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:23.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T17:42:28.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-03-04T17:42:28.485 DEBUG:boto:path=/sbnvju-70/ 2024-03-04T17:42:28.485 DEBUG:boto:auth_path=/sbnvju-70/ 2024-03-04T17:42:28.486 DEBUG:boto:Method: PUT 2024-03-04T17:42:28.486 DEBUG:boto:Path: /sbnvju-70/ 2024-03-04T17:42:28.486 DEBUG:boto:Data: 2024-03-04T17:42:28.486 DEBUG:boto:Headers: {} 2024-03-04T17:42:28.486 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:42:28.486 DEBUG:boto:Port: 8000 2024-03-04T17:42:28.486 DEBUG:boto:Params: {} 2024-03-04T17:42:28.487 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:42:28.487 DEBUG:boto:Token: None 2024-03-04T17:42:28.487 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:42:28 GMT /sbnvju-70/ 2024-03-04T17:42:28.487 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:PfkHByoH3kmkVOqRjY27dxHRKjg= 2024-03-04T17:42:28.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:42:28 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PfkHByoH3kmkVOqRjY27dxHRKjg=', 'Content-Length': '0'} 2024-03-04T17:42:28.510 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b5285011feb9b34-0065e60804-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:42:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:42:28.510 DEBUG:rgw_multi.tests:created bucket=sbnvju-70 2024-03-04T17:42:28.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:28.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:28.511 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:28.568 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:28.568 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:28.626 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:42:28.647 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709574148.508071_969.1","last_update":"2024-03-04T17:42:28.508071Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:42:28.647 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:28.647 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:42:28.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:28.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:28.648 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:28.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:28.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:28.780+0000 7f60eac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:28.781+0000 7f60eac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:28.805 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:28Z"} 2024-03-04T17:42:28.806 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:28Z"} 2024-03-04T17:42:28.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:28.806 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:28.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:28.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T17:42:33.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:33.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:33.808 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:33.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:33.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:33.907 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:33.906+0000 7f1a0b044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:33.908 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:33.907+0000 7f1a0b044800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:33.930 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:33Z"} 2024-03-04T17:42:33.930 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:33Z"} 2024-03-04T17:42:33.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:33.930 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:33.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:33.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T17:42:38.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:38.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:38.932 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:38.979 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:38.979 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:39.041 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:39.040+0000 7fb2f0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:39.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:39.041+0000 7fb2f0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:39.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:39Z"} 2024-03-04T17:42:39.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:39Z"} 2024-03-04T17:42:39.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:39.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:39.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:39.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.613615_561.1 2024-03-04T17:42:44.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:44.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:44.067 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:44.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:44.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:44.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:44.181+0000 7fdb4e7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:44.183 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:44.182+0000 7fdb4e7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:44.206 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:44Z"} 2024-03-04T17:42:44.207 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:44Z"} 2024-03-04T17:42:44.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:44.207 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:44.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:44.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.890340_643.1 2024-03-04T17:42:49.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:49.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:49.209 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:49.256 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:49.257 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:49.317 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:49.317+0000 7f4765630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:49.317 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:49.318+0000 7f4765630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:49.342 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:49Z"} 2024-03-04T17:42:49.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:49Z"} 2024-03-04T17:42:49.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:49.342 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:49.342 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:49.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:42:54.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:54.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:54.344 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:54.395 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:54.395 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:54.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:54.446+0000 7f899f617800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:54.447 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:54.446+0000 7f899f617800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:54.470 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:54Z"} 2024-03-04T17:42:54.470 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:54Z"} 2024-03-04T17:42:54.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:54.470 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:54.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:54.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571980.296697_734.1 2024-03-04T17:42:59.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:59.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:42:59.472 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:42:59.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:42:59.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:42:59.588 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:59.588+0000 7f6519d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:59.589 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:42:59.588+0000 7f6519d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:42:59.614 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:59Z"} 2024-03-04T17:42:59.614 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:59Z"} 2024-03-04T17:42:59.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:42:59.614 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:59.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:42:59.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T17:43:04.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:04.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:04.616 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:04.690 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:04.776 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:04.775+0000 7f42b8fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:04.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:04.776+0000 7f42b8fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:04.799 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:04Z"} 2024-03-04T17:43:04.800 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:04Z"} 2024-03-04T17:43:04.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:04.800 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:04.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:04.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.608330_559.1 2024-03-04T17:43:09.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:09.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:09.802 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:09.851 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:09.851 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:09.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:09.918+0000 7f5045d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:09.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:09.919+0000 7f5045d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:09.943 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:09Z"} 2024-03-04T17:43:09.943 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:09Z"} 2024-03-04T17:43:09.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:09.943 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:09.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:09.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.640587_564.1 2024-03-04T17:43:14.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:14.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:14.945 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:14.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:14.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:15.066 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:15.065+0000 7fcd88a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:15.066 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:15.066+0000 7fcd88a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:15.093 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:15Z"} 2024-03-04T17:43:15.093 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:15Z"} 2024-03-04T17:43:15.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:15.094 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:15.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:15.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T17:43:20.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:20.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:20.095 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:20.172 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:20.172 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:20.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:20.266+0000 7f19b6342800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:20.267+0000 7f19b6342800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:20.299 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:20Z"} 2024-03-04T17:43:20.299 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:20Z"} 2024-03-04T17:43:20.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:20.300 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:20.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:20.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T17:43:25.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:25.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:25.302 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:25.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:25.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:25.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:25.403+0000 7f485a931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:25.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:25.404+0000 7f485a931800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:25.430 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:25Z"} 2024-03-04T17:43:25.431 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:25Z"} 2024-03-04T17:43:25.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:25.431 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:25.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:25.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T17:43:30.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:30.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:30.432 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:30.545 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:30.544+0000 7fe1a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:30.546 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:30.545+0000 7fe1a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:30.577 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:30Z"} 2024-03-04T17:43:30.577 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:30Z"} 2024-03-04T17:43:30.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:30.578 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:30.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:30.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T17:43:35.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:35.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:35.579 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:35.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:35.694+0000 7fcca8d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:35.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:35.695+0000 7fcca8d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:35.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:35Z"} 2024-03-04T17:43:35.727 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:35Z"} 2024-03-04T17:43:35.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:35.727 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:35.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:35.727 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:43:40.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:40.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:40.728 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:40.775 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:40.776 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:40.833 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:40.832+0000 7f64f9726800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:40.833 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:40.833+0000 7f64f9726800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:40.857 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:40Z"} 2024-03-04T17:43:40.857 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:40Z"} 2024-03-04T17:43:40.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:40.857 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:40.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:40.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:43:45.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:45.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:45.859 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:45.907 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:45.907 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:45.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:45.958+0000 7fa1ab411800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:45.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:45.959+0000 7fa1ab411800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:45.982 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:45Z"} 2024-03-04T17:43:45.982 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:45Z"} 2024-03-04T17:43:45.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:45.982 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:45.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:45.983 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T17:43:50.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:50.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:50.984 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:51.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:51.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:51.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:51.085+0000 7fed23726800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:51.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:51.086+0000 7fed23726800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:51.110 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:51Z"} 2024-03-04T17:43:51.110 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:51Z"} 2024-03-04T17:43:51.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:51.111 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:51.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:51.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.613615_561.1 2024-03-04T17:43:56.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:56.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:43:56.113 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:43:56.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:43:56.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:43:56.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:56.229+0000 7f1b5de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:56.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:43:56.230+0000 7f1b5de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:43:56.253 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:56Z"} 2024-03-04T17:43:56.253 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:56Z"} 2024-03-04T17:43:56.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:43:56.254 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:56.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:43:56.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571978.220171_727.1 2024-03-04T17:44:01.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:01.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:01.255 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:01.308 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:01.308 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:01.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:01.381+0000 7f0cb5533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:01.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:01.382+0000 7f0cb5533800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:01.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:01Z"} 2024-03-04T17:44:01.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:01Z"} 2024-03-04T17:44:01.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:01.414 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:01.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:01.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:44:06.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:06.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:06.416 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:06.469 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:06.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:06.527+0000 7fd55b12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:06.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:06.528+0000 7fd55b12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:06.551 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:06Z"} 2024-03-04T17:44:06.551 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:06Z"} 2024-03-04T17:44:06.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:06.552 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:06.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:06.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T17:44:11.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:11.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:11.554 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:11.607 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:11.607 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:11.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:11.662+0000 7f7dcf015800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:11.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:11.663+0000 7f7dcf015800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:11.687 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:11Z"} 2024-03-04T17:44:11.688 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:11Z"} 2024-03-04T17:44:11.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:11.688 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:11.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:11.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T17:44:16.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:16.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:16.689 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:16.742 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:16.743 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:16.805 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:16.805+0000 7f9e86e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:16.806 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:16.805+0000 7f9e86e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:16.829 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:16Z"} 2024-03-04T17:44:16.830 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:16Z"} 2024-03-04T17:44:16.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:16.830 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:16.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:16.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T17:44:21.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:21.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:21.832 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:21.916 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:21.916 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:22.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:22.024+0000 7ff3f9e85800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:22.029 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:22.029+0000 7ff3f9e85800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:22.077 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:22Z"} 2024-03-04T17:44:22.077 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:22Z"} 2024-03-04T17:44:22.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:22.077 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:22.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:22.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.890340_643.1 2024-03-04T17:44:27.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:27.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:27.079 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:27.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:27.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:27.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:27.221+0000 7f2993526800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:27.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:27.227+0000 7f2993526800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:27.318 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:27Z"} 2024-03-04T17:44:27.318 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:27Z"} 2024-03-04T17:44:27.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:27.319 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:27.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:27.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T17:44:32.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:32.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:32.321 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:32.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:32.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:32.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:32.438+0000 7fd3eda24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:32.439 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:32.438+0000 7fd3eda24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:32.461 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:32Z"} 2024-03-04T17:44:32.462 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:32Z"} 2024-03-04T17:44:32.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:32.462 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:32.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:32.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:44:37.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:37.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:37.464 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:37.510 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:37.510 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:37.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:37.568+0000 7f45d988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:37.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:37.569+0000 7f45d988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:37.603 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:37Z"} 2024-03-04T17:44:37.603 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:37Z"} 2024-03-04T17:44:37.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:37.603 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:37.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:37.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T17:44:42.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:42.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:42.605 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:42.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:42.651 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:42.701+0000 7fb0d3011800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:42.702+0000 7fb0d3011800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:42.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:42Z"} 2024-03-04T17:44:42.728 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:42Z"} 2024-03-04T17:44:42.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:42.729 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:42.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:42.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T17:44:47.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:47.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:47.730 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:47.776 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:47.776 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:47.832+0000 7fc5ad92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:47.833+0000 7fc5ad92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:47.856 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:47Z"} 2024-03-04T17:44:47.856 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:47Z"} 2024-03-04T17:44:47.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:47.857 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:47.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:47.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T17:44:52.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:52.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:52.858 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:52.907 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:52.907 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:52.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:52.957+0000 7f5ac1731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:52.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:52.958+0000 7f5ac1731800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:52.984 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:52Z"} 2024-03-04T17:44:52.984 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:52Z"} 2024-03-04T17:44:52.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:52.985 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:52.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:52.985 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:44:57.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:57.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:44:57.987 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:44:58.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:44:58.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:44:58.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:58.092+0000 7fba8bc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:58.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:44:58.093+0000 7fba8bc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:44:58.118 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:58Z"} 2024-03-04T17:44:58.118 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:58Z"} 2024-03-04T17:44:58.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:44:58.119 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:58.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:44:58.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.881246_635.1 2024-03-04T17:45:03.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:03.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:03.121 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:03.228+0000 7f9d9e067800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:03.229+0000 7f9d9e067800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:03.252 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:03Z"} 2024-03-04T17:45:03.252 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:03Z"} 2024-03-04T17:45:03.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:03.252 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:03.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:03.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T17:45:08.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:08.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:08.254 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:08.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:08.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:08.359 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:08.358+0000 7f3378e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:08.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:08.359+0000 7f3378e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:08.384 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:08Z"} 2024-03-04T17:45:08.384 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:08Z"} 2024-03-04T17:45:08.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:08.385 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:08.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:08.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571977.996460_704.1 2024-03-04T17:45:13.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:13.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:13.386 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:13.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:13.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:13.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:13.495+0000 7fa2e0c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:13.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:13.496+0000 7fa2e0c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:13.519 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:13Z"} 2024-03-04T17:45:13.519 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:13Z"} 2024-03-04T17:45:13.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:13.519 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:13.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:13.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:45:18.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:18.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:18.521 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:18.569 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:18.569 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:18.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:18.637+0000 7fbad8630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:18.639 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:18.639+0000 7fbad8630800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:18.671 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:18Z"} 2024-03-04T17:45:18.671 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:18Z"} 2024-03-04T17:45:18.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:18.671 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:18.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:18.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T17:45:23.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:23.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:23.673 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:23.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:23.723 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:23.775 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:23.775+0000 7fa72ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:23.776 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:23.776+0000 7fa72ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:23.799 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:23Z"} 2024-03-04T17:45:23.799 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:23Z"} 2024-03-04T17:45:23.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:23.800 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:23.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:23.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.640587_564.1 2024-03-04T17:45:28.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:28.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:28.802 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:28.853 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:28.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:28.921 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:28.920+0000 7f4286b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:28.922 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:28.921+0000 7f4286b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:28.947 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:28Z"} 2024-03-04T17:45:28.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:28Z"} 2024-03-04T17:45:28.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:28.947 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:28.947 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:28.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T17:45:33.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:33.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:33.949 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:34.071 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:34.071+0000 7fda4531e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:34.072 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:34.072+0000 7fda4531e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:34.098 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:34Z"} 2024-03-04T17:45:34.098 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:34Z"} 2024-03-04T17:45:34.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:34.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:34.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:34.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.692350_570.1 2024-03-04T17:45:39.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:39.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:39.101 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:39.150 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:39.150 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:39.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:39.223+0000 7f7e70214800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:39.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:39.224+0000 7f7e70214800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:39.288 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:39Z"} 2024-03-04T17:45:39.288 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:39Z"} 2024-03-04T17:45:39.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:39.289 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:39.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:39.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.641846_565.1 2024-03-04T17:45:44.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:44.291 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:44.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:44.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:44.421+0000 7f49bd430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:44.422+0000 7f49bd430800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:44.487 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T17:45:44.487 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T17:45:44.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:44.487 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:44.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:44.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:45:49.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:49.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:49.557 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:49.557 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:49.724 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:49.724+0000 7febaaa10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.725 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:49.725+0000 7febaaa10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:49.767 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.767 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T17:45:49.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:49.768 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:49.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:49.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571977.828757_695.1 2024-03-04T17:45:54.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:54.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:54.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:45:54.868 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:54.867+0000 7f0688644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:54.868 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:45:54.868+0000 7f0688644800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:45:54.892 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T17:45:54.892 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T17:45:54.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:45:54.892 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:54.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:45:54.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.608330_559.1 2024-03-04T17:45:59.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:59.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:45:59.894 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:45:59.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:45:59.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:00.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:00.003+0000 7f8417e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:00.004 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:00.004+0000 7f8417e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:00.029 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:00Z"} 2024-03-04T17:46:00.029 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:00Z"} 2024-03-04T17:46:00.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:00.029 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:00.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:00.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T17:46:05.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:05.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:05.032 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:05.078 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:05.078 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:05.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:05.142+0000 7fddd7c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:05.143 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:05.142+0000 7fddd7c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:05.168 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:05Z"} 2024-03-04T17:46:05.168 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:05Z"} 2024-03-04T17:46:05.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:05.169 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:05.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:05.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T17:46:10.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:10.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:10.170 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:10.217 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:10.217 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:10.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:10.275+0000 7fde5882e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:10.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:10.276+0000 7fde5882e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:10.299 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:10Z"} 2024-03-04T17:46:10.299 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:10Z"} 2024-03-04T17:46:10.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:10.300 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:10.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:10.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T17:46:15.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:15.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:15.301 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:15.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:15.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:15.398+0000 7fdeba69d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:15.399+0000 7fdeba69d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:15.422 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:15Z"} 2024-03-04T17:46:15.422 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:15Z"} 2024-03-04T17:46:15.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:15.423 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:15.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:15.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:46:20.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:20.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:20.425 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:20.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:20.564+0000 7f7bbed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:20.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:20.565+0000 7f7bbed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:20.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:20Z"} 2024-03-04T17:46:20.609 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:20Z"} 2024-03-04T17:46:20.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:20.609 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:20.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:20.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:46:25.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:25.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:25.612 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:25.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:25.659 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:25.726 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:25.726+0000 7f3e89ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:25.726 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:25.726+0000 7f3e89ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:25.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:25Z"} 2024-03-04T17:46:25.758 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:25Z"} 2024-03-04T17:46:25.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:25.758 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:25.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:25.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:46:30.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:30.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:30.759 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:30.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:30.879+0000 7fec1ad31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:30.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:30.880+0000 7fec1ad31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:30.910 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:30Z"} 2024-03-04T17:46:30.910 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:30Z"} 2024-03-04T17:46:30.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:30.911 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:30.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:30.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T17:46:35.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:35.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:35.912 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:35.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:35.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:36.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:36.023+0000 7fd6a0c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:36.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:36.024+0000 7fd6a0c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:36.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:36Z"} 2024-03-04T17:46:36.049 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:36Z"} 2024-03-04T17:46:36.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:36.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:36.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:36.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T17:46:41.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:41.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:41.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:41.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:41.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:41.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:41.149+0000 7f5c32c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:41.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:41.150+0000 7f5c32c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:41.173 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:41Z"} 2024-03-04T17:46:41.173 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:41Z"} 2024-03-04T17:46:41.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:41.174 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:41.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:41.174 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T17:46:46.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:46.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:46.176 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:46.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:46.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:46.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:46.292+0000 7ff415c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:46.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:46.293+0000 7ff415c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:46.325 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:46Z"} 2024-03-04T17:46:46.325 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:46Z"} 2024-03-04T17:46:46.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:46.326 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:46.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:46.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T17:46:51.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:51.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:51.328 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:51.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:51.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:51.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:51.435+0000 7f6d6db2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:51.436 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:51.435+0000 7f6d6db2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:51.460 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:51Z"} 2024-03-04T17:46:51.460 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:51Z"} 2024-03-04T17:46:51.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:51.460 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:51.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:51.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T17:46:56.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:56.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:46:56.462 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:46:56.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:46:56.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:46:56.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:56.562+0000 7f7e44b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:56.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:46:56.563+0000 7f7e44b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:46:56.588 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:56Z"} 2024-03-04T17:46:56.589 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:56Z"} 2024-03-04T17:46:56.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:46:56.589 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:56.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:46:56.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T17:47:01.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:01.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:01.591 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:01.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:01.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:01.699 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:01.699+0000 7fea35830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:01.700 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:01.700+0000 7fea35830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:01.723 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:01Z"} 2024-03-04T17:47:01.724 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:01Z"} 2024-03-04T17:47:01.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:01.724 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:01.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:01.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T17:47:06.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:06.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:06.726 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:06.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:06.822+0000 7fe7dea11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:06.823 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:06.823+0000 7fe7dea11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:06.845 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:06Z"} 2024-03-04T17:47:06.845 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:06Z"} 2024-03-04T17:47:06.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:06.845 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:06.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:06.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T17:47:11.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:11.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:11.847 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:11.896 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:11.955 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:11.955+0000 7fb946d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:11.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:11.955+0000 7fb946d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:11.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:11Z"} 2024-03-04T17:47:11.986 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:11Z"} 2024-03-04T17:47:11.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:11.987 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:11.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:11.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T17:47:16.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:16.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:16.989 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:17.042 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:17.042 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:17.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:17.104+0000 7f9da5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:17.105 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:17.105+0000 7f9da5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:17.128 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:17Z"} 2024-03-04T17:47:17.128 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:17Z"} 2024-03-04T17:47:17.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:17.129 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:17.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:17.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T17:47:22.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:22.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:22.131 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:22.181 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:22.181 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:22.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:22.243+0000 7f2232444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:22.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:22.244+0000 7f2232444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:22.266 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:22Z"} 2024-03-04T17:47:22.267 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:22Z"} 2024-03-04T17:47:22.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:22.267 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:22.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:22.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571978.120400_708.1 2024-03-04T17:47:27.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:27.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:27.269 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:27.317 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:27.318 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:27.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:27.368+0000 7fbd83415800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:27.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:27.369+0000 7fbd83415800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:27.393 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:27Z"} 2024-03-04T17:47:27.393 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:27Z"} 2024-03-04T17:47:27.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:27.394 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:27.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:27.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T17:47:32.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:32.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:32.395 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:32.443 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:32.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:32.493+0000 7fe567842800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:32.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:32.494+0000 7fe567842800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:32.515 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:32Z"} 2024-03-04T17:47:32.515 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:32Z"} 2024-03-04T17:47:32.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:32.515 DEBUG:rgw_multi.tests:log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:32.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:32.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T17:47:37.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-03-04T17:47:37.521 DEBUG:boto:path=/sbnvju-71/ 2024-03-04T17:47:37.521 DEBUG:boto:auth_path=/sbnvju-71/ 2024-03-04T17:47:37.521 DEBUG:boto:Method: PUT 2024-03-04T17:47:37.521 DEBUG:boto:Path: /sbnvju-71/ 2024-03-04T17:47:37.522 DEBUG:boto:Data: 2024-03-04T17:47:37.522 DEBUG:boto:Headers: {} 2024-03-04T17:47:37.522 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:47:37.522 DEBUG:boto:Port: 8000 2024-03-04T17:47:37.522 DEBUG:boto:Params: {} 2024-03-04T17:47:37.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:47:37.523 DEBUG:boto:Token: None 2024-03-04T17:47:37.523 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:47:37 GMT /sbnvju-71/ 2024-03-04T17:47:37.523 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d48wkmXAtCIOizaBXHoP30ObX/w= 2024-03-04T17:47:37.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:47:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d48wkmXAtCIOizaBXHoP30ObX/w=', 'Content-Length': '0'} 2024-03-04T17:47:37.546 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062e6309f616d18a5-0065e60939-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:47:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:47:37.546 DEBUG:rgw_multi.tests:created bucket=sbnvju-71 2024-03-04T17:47:37.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:37.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:37.547 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:37.599 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:37.599 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:47:37.672 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709574457.544617_986.1","last_update":"2024-03-04T17:47:37.544617Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:47:37.672 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:37.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:47:37.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:37.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:37.673 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:37.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:37.749 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:37.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:37.896+0000 7f681d815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:37.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:37.896+0000 7f681d815800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:37.927 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:37Z"} 2024-03-04T17:47:37.927 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:37Z"} 2024-03-04T17:47:37.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:37.928 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:37.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:37.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:47:42.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:42.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:42.929 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:43.020 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:43.020 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:43.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:43.121+0000 7f1f7852f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:43.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:43.122+0000 7f1f7852f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:43.146 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:43Z"} 2024-03-04T17:47:43.146 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:43Z"} 2024-03-04T17:47:43.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:43.146 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:43.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:43.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T17:47:48.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:48.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:48.149 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:48.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:48.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:48.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:48.279+0000 7fe0e9e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:48.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:48.280+0000 7fe0e9e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:48.308 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:48Z"} 2024-03-04T17:47:48.308 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:48Z"} 2024-03-04T17:47:48.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:48.308 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:48.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:48.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:47:53.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:53.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:53.310 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:53.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:53.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:53.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:53.416+0000 7f1c7a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:53.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:53.417+0000 7f1c7a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:53.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:53Z"} 2024-03-04T17:47:53.442 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:53Z"} 2024-03-04T17:47:53.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:53.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:53.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:53.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.886755_642.1 2024-03-04T17:47:58.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:58.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:47:58.444 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:47:58.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:47:58.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:47:58.544 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:58.544+0000 7fd25cd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:58.545 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:47:58.545+0000 7fd25cd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:47:58.566 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:58Z"} 2024-03-04T17:47:58.567 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:58Z"} 2024-03-04T17:47:58.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:47:58.567 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:58.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:47:58.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:48:03.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:03.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:03.568 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:03.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:03.675+0000 7fbaf832c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:03.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:03.676+0000 7fbaf832c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:03.703 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:03Z"} 2024-03-04T17:48:03.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:03Z"} 2024-03-04T17:48:03.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:03.704 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:03.704 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:03.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:48:08.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:08.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:08.705 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:08.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:08.753 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:08.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:08.815+0000 7fd88392b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:08.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:08.816+0000 7fd88392b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:08.843 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:08Z"} 2024-03-04T17:48:08.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:08Z"} 2024-03-04T17:48:08.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:08.844 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:08.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:08.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:48:13.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:13.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:13.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:13.894 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:13.894 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:13.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:13.966+0000 7f7784012800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:13.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:13.967+0000 7f7784012800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:13.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:13Z"} 2024-03-04T17:48:13.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:13Z"} 2024-03-04T17:48:13.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:13.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:13.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:13.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:48:18.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:18.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:18.997 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:19.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:19.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:19.106 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:19.106+0000 7f7a0f54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:19.107 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:19.107+0000 7f7a0f54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:19.140 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:19Z"} 2024-03-04T17:48:19.140 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:19Z"} 2024-03-04T17:48:19.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:19.141 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:19.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:19.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:48:24.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:24.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:24.143 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:24.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:24.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:24.261+0000 7f8817e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:24.262+0000 7f8817e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:24.288 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:24Z"} 2024-03-04T17:48:24.289 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:24Z"} 2024-03-04T17:48:24.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:24.289 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:24.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:24.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.175444_719.1 2024-03-04T17:48:29.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:29.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:29.291 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:29.340 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:29.392+0000 7fe2ebb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:29.393+0000 7fe2ebb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:29.421 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:29Z"} 2024-03-04T17:48:29.421 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:29Z"} 2024-03-04T17:48:29.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:29.422 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:29.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:29.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:48:34.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:34.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:34.424 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:34.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:34.472 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:34.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:34.525+0000 7f20eef26800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:34.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:34.526+0000 7f20eef26800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:34.552 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:34Z"} 2024-03-04T17:48:34.553 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:34Z"} 2024-03-04T17:48:34.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:34.553 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:34.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:34.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:48:39.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:39.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:39.554 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:39.606 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:39.607 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:39.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:39.664+0000 7f73bcf33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:39.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:39.665+0000 7f73bcf33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:39.714 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:39Z"} 2024-03-04T17:48:39.714 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:39Z"} 2024-03-04T17:48:39.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:39.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:39.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:39.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:48:44.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:44.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:44.716 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:44.767 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:44.767 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:44.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:44.931+0000 7f5ad3024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:44.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:44.935+0000 7f5ad3024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:44.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:44Z"} 2024-03-04T17:48:44.981 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:44Z"} 2024-03-04T17:48:44.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:44.982 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:44.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:44.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:48:49.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:49.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:49.984 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:50.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:50.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:50.106 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:50.105+0000 7f0b1fec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:50.106 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:50.106+0000 7f0b1fec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:50.133 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:50Z"} 2024-03-04T17:48:50.133 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:50Z"} 2024-03-04T17:48:50.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:50.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:50.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:50.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:48:55.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:55.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:48:55.135 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:48:55.181 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:48:55.182 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:48:55.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:55.243+0000 7f6c00d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:55.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:48:55.244+0000 7f6c00d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:48:55.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:55Z"} 2024-03-04T17:48:55.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:55Z"} 2024-03-04T17:48:55.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:48:55.274 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:55.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:48:55.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:49:00.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:00.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:00.276 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:00.326 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:00.326 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:00.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:00.388+0000 7fda2a214800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:00.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:00.389+0000 7fda2a214800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:00.417 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:00Z"} 2024-03-04T17:49:00.417 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:00Z"} 2024-03-04T17:49:00.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:00.417 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:00.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:00.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:49:05.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:05.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:05.419 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:05.465 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:05.466 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:05.514 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:05.514+0000 7f5335337800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:05.515 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:05.515+0000 7f5335337800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:05.537 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:05Z"} 2024-03-04T17:49:05.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:05Z"} 2024-03-04T17:49:05.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:05.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:05.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:05.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:49:10.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:10.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:10.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:10.586 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:10.587 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:10.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:10.644+0000 7fecdb51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:10.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:10.645+0000 7fecdb51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:10.674 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:10Z"} 2024-03-04T17:49:10.675 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:10Z"} 2024-03-04T17:49:10.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:10.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:10.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:10.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:49:15.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:15.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:15.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:15.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:15.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:15.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:15.787+0000 7f3c2a72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:15.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:15.788+0000 7f3c2a72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:15.813 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:15Z"} 2024-03-04T17:49:15.813 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:15Z"} 2024-03-04T17:49:15.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:15.813 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:15.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:15.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.692350_570.1 2024-03-04T17:49:20.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:20.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:20.815 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:20.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:20.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:20.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:20.928+0000 7f124ce24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:20.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:20.929+0000 7f124ce24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:20.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:20Z"} 2024-03-04T17:49:20.959 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:20Z"} 2024-03-04T17:49:20.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:20.959 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:20.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:20.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:49:25.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:25.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:25.961 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:26.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:26.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:26.063 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:26.063+0000 7f4808d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:26.064 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:26.064+0000 7f4808d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:26.086 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:26Z"} 2024-03-04T17:49:26.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:26Z"} 2024-03-04T17:49:26.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:26.087 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:26.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:26.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T17:49:31.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:31.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:31.089 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:31.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:31.136 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:31.184 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:31.184+0000 7f840c52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:31.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:31.185+0000 7f840c52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:31.207 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:31Z"} 2024-03-04T17:49:31.207 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:31Z"} 2024-03-04T17:49:31.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:31.208 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:31.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:31.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T17:49:36.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:36.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:36.210 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:36.256 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:36.256 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:36.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:36.333+0000 7f7fbb414800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:36.334 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:36.334+0000 7f7fbb414800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:36.357 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:36Z"} 2024-03-04T17:49:36.357 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:36Z"} 2024-03-04T17:49:36.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:36.357 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:36.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:36.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:49:41.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:41.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:41.359 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:41.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:41.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:41.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:41.473+0000 7efcebb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:41.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:41.474+0000 7efcebb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:41.497 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:41Z"} 2024-03-04T17:49:41.498 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:41Z"} 2024-03-04T17:49:41.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:41.498 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:41.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:41.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:49:46.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:46.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:46.500 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:46.547 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:46.547 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:46.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:46.605+0000 7f98dda44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:46.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:46.606+0000 7f98dda44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:46.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:46Z"} 2024-03-04T17:49:46.629 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:46Z"} 2024-03-04T17:49:46.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:46.629 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:46.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:46.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T17:49:51.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:51.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:51.631 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:51.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:51.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:51.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:51.738+0000 7f1ea7b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:51.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:51.740+0000 7f1ea7b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:51.782 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:51Z"} 2024-03-04T17:49:51.783 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:51Z"} 2024-03-04T17:49:51.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:51.783 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:51.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:51.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:49:56.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:56.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:49:56.784 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:49:56.836 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:49:56.836 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:49:56.909 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:56.909+0000 7f420472f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:56.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:49:56.910+0000 7f420472f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:49:56.938 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572600.174134_809.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:16:40.174134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:56Z"} 2024-03-04T17:49:56.938 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572600.174134_809.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:16:40.174134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:56Z"} 2024-03-04T17:49:56.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572600.174134_809.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:16:40.174134Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:49:56.939 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:56.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1709572600.174134_809.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:49:56.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709572600.174134_809.1 2024-03-04T17:50:01.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:01.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:01.940 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:01.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:01.990 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:02.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:02.042+0000 7fa77dd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:02.043 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:02.043+0000 7fa77dd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:02.074 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:02Z"} 2024-03-04T17:50:02.074 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:02Z"} 2024-03-04T17:50:02.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:02.074 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:02.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:02.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:50:07.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:07.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:07.076 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:07.129 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:07.130 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:07.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:07.194+0000 7fc37f6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:07.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:07.195+0000 7fc37f6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:07.226 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:07Z"} 2024-03-04T17:50:07.227 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:07Z"} 2024-03-04T17:50:07.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:07.227 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:07.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:07.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:50:12.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:12.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:12.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:12.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:12.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:12.335 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:12.335+0000 7f5d9f522800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:12.335 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:12.336+0000 7f5d9f522800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:12.358 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:12Z"} 2024-03-04T17:50:12.358 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:12Z"} 2024-03-04T17:50:12.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:12.359 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:12.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:12.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:50:17.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:17.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:17.360 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:17.409 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:17.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:17.476 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:17.476+0000 7f98c814b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:17.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:17.477+0000 7f98c814b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:17.502 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:17Z"} 2024-03-04T17:50:17.502 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:17Z"} 2024-03-04T17:50:17.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:17.502 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:17.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:17.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:50:22.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:22.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:22.504 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:22.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:22.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:22.607 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:22.607+0000 7f68fc62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:22.607 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:22.608+0000 7f68fc62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:22.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:22Z"} 2024-03-04T17:50:22.630 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:22Z"} 2024-03-04T17:50:22.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:22.631 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:22.631 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:22.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:50:27.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:27.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:27.632 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:27.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:27.702 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:27.851 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:27.851+0000 7f787ae10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:27.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:27.852+0000 7f787ae10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:27.879 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:27Z"} 2024-03-04T17:50:27.879 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:27Z"} 2024-03-04T17:50:27.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:27.879 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:27.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:27.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:50:32.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:32.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:32.882 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:33.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:33.017+0000 7f2cb9b20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:33.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:33.018+0000 7f2cb9b20800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:33.045 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:33Z"} 2024-03-04T17:50:33.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:33Z"} 2024-03-04T17:50:33.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:33.046 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:33.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:33.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.838159_698.1 2024-03-04T17:50:38.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:38.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:38.048 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:38.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:38.185+0000 7f2f5ad0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:38.186+0000 7f2f5ad0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:38.214 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T17:50:38.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T17:50:38.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:38.214 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:38.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:38.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T17:50:43.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:43.216 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:43.263 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:43.264 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:43.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:43.327+0000 7f56768c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:43.328+0000 7f56768c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:43.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.351 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T17:50:43.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:43.351 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:43.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:43.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T17:50:48.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:48.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:48.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:48.402 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:48.402 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:48.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:48.470+0000 7fc841a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:48.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:48.471+0000 7fc841a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:48.508 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:48Z"} 2024-03-04T17:50:48.508 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:48Z"} 2024-03-04T17:50:48.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:48.508 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:48.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:48.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:50:53.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:53.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:53.511 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:53.557 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:53.557 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:53.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:53.624+0000 7f7f2992f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:53.625 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:53.625+0000 7f7f2992f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:53.647 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:53Z"} 2024-03-04T17:50:53.647 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:53Z"} 2024-03-04T17:50:53.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:53.648 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:53.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:53.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:50:58.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:58.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:50:58.650 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:50:58.696 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:50:58.696 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:50:58.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:58.760+0000 7fa4b6d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:58.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:50:58.761+0000 7fa4b6d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:50:58.786 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:58Z"} 2024-03-04T17:50:58.786 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:58Z"} 2024-03-04T17:50:58.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:50:58.787 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:58.787 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:50:58.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:51:03.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:03.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:03.788 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:03.834 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:03.834 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:03.882 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:03.882+0000 7f0dd9013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:03.882 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:03.883+0000 7f0dd9013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:03.905 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:03Z"} 2024-03-04T17:51:03.906 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:03Z"} 2024-03-04T17:51:03.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:03.906 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:03.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:03.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.828757_695.1 2024-03-04T17:51:08.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:08.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:08.908 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:08.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:08.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:09.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:09.021+0000 7f6737213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:09.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:09.022+0000 7f6737213800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:09.044 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:09Z"} 2024-03-04T17:51:09.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:09Z"} 2024-03-04T17:51:09.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:09.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:09.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:09.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T17:51:14.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:14.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:14.047 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:14.094 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:14.094 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:14.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:14.151+0000 7fa77fcc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:14.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:14.151+0000 7fa77fcc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:14.176 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:14Z"} 2024-03-04T17:51:14.176 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:14Z"} 2024-03-04T17:51:14.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:14.176 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:14.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:14.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.838159_698.1 2024-03-04T17:51:19.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:19.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:19.178 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:19.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:19.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:19.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:19.288+0000 7ff874a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:19.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:19.288+0000 7ff874a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:19.311 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:19Z"} 2024-03-04T17:51:19.311 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:19Z"} 2024-03-04T17:51:19.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:19.312 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:19.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:19.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:51:24.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:24.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:24.314 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:24.360 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:24.361 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:24.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:24.411+0000 7fe55f122800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:24.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:24.412+0000 7fe55f122800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:24.435 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:24Z"} 2024-03-04T17:51:24.435 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:24Z"} 2024-03-04T17:51:24.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:24.435 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:24.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:24.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T17:51:29.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:29.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:29.437 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:29.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:29.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:29.534 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:29.534+0000 7f7f709a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:29.535 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:29.535+0000 7f7f709a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:29.557 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:29Z"} 2024-03-04T17:51:29.558 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:29Z"} 2024-03-04T17:51:29.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:29.558 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:29.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:29.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:51:34.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:34.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:34.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:34.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:34.660+0000 7f61107a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:34.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:34.661+0000 7f61107a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:34.682 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:34Z"} 2024-03-04T17:51:34.683 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:34Z"} 2024-03-04T17:51:34.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:34.683 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:34.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:34.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.616944_562.1 2024-03-04T17:51:39.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:39.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:39.685 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:39.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:39.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:39.782 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:39.782+0000 7f801d02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:39.783 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:39.783+0000 7f801d02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:39.806 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:39Z"} 2024-03-04T17:51:39.806 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:39Z"} 2024-03-04T17:51:39.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:39.806 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:39.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:39.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.220171_727.1 2024-03-04T17:51:44.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:44.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:44.808 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:44.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:44.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:44.927 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:44.927+0000 7f71b9230800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:44.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:44.928+0000 7f71b9230800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:44.956 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:44Z"} 2024-03-04T17:51:44.956 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:44Z"} 2024-03-04T17:51:44.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:44.957 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:44.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:44.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T17:51:49.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:49.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:49.959 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:50.006 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:50.006 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:50.071 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:50.071+0000 7f2219b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:50.072 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:50.072+0000 7f2219b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:50.103 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:50Z"} 2024-03-04T17:51:50.103 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:50Z"} 2024-03-04T17:51:50.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:50.103 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:50.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:50.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:51:55.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:55.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:51:55.105 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:51:55.151 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:51:55.151 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:51:55.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:55.199+0000 7fe33d08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:51:55.200+0000 7fe33d08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:51:55.222 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:55Z"} 2024-03-04T17:51:55.223 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:55Z"} 2024-03-04T17:51:55.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:51:55.223 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:55.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:51:55.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:52:00.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:00.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:00.225 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:00.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:00.330+0000 7ff2242c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:00.331 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:00.331+0000 7ff2242c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:00.362 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:00Z"} 2024-03-04T17:52:00.362 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:00Z"} 2024-03-04T17:52:00.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:00.362 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:00.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:00.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.116657_707.1 2024-03-04T17:52:05.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:05.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:05.365 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:05.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:05.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:05.480 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:05.480+0000 7fd299d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:05.481 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:05.481+0000 7fd299d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:05.509 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:05Z"} 2024-03-04T17:52:05.510 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:05Z"} 2024-03-04T17:52:05.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:05.510 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:05.510 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:05.510 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:52:10.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:10.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:10.511 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:10.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:10.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:10.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:10.615+0000 7f4d79c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:10.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:10.616+0000 7f4d79c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:10.641 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:10Z"} 2024-03-04T17:52:10.641 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:10Z"} 2024-03-04T17:52:10.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:10.641 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:10.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:10.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:52:15.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:15.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:15.643 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:15.696 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:15.696 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:15.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:15.767+0000 7fab3c71e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:15.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:15.768+0000 7fab3c71e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:15.795 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:15Z"} 2024-03-04T17:52:15.795 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:15Z"} 2024-03-04T17:52:15.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:15.796 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:15.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:15.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:52:20.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:20.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:20.798 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:20.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:20.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:20.893 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:20.893+0000 7f73c542e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:20.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:20.894+0000 7f73c542e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:20.917 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:20Z"} 2024-03-04T17:52:20.917 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:20Z"} 2024-03-04T17:52:20.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:20.918 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:20.918 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:20.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:52:25.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:25.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:25.920 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:25.968 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:25.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:26.034 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:26.034+0000 7f24b0013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:26.035 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:26.035+0000 7f24b0013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:26.058 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:26Z"} 2024-03-04T17:52:26.058 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:26Z"} 2024-03-04T17:52:26.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:26.058 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:26.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:26.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:52:31.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:31.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:31.060 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:31.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:31.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:31.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:31.177+0000 7f38e1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:31.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:31.178+0000 7f38e1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:31.209 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:31Z"} 2024-03-04T17:52:31.209 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:31Z"} 2024-03-04T17:52:31.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:31.210 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:31.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:31.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T17:52:36.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:36.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:36.212 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:36.321 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:36.321+0000 7f9c5ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:36.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:36.322+0000 7f9c5ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:36.355 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:36Z"} 2024-03-04T17:52:36.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:36Z"} 2024-03-04T17:52:36.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:36.355 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:36.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:36.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:52:41.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:41.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:41.357 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:41.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:41.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:41.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:41.459+0000 7f45ebc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:41.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:41.460+0000 7f45ebc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:41.485 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:41Z"} 2024-03-04T17:52:41.486 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:41Z"} 2024-03-04T17:52:41.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:41.486 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:41.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:41.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:52:46.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2024-03-04T17:52:46.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:46.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:46.490 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:46.544 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:46.544 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:46.598 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:52:46.619 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709574457.544617_986.1","last_update":"2024-03-04T17:47:37.544617Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:52:46.619 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:46.620 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:52:46.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:46.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:46.620 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:46.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:46.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:46.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:46.738+0000 7f20b2f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:46.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:46.739+0000 7f20b2f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:46.768 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:46Z"} 2024-03-04T17:52:46.768 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:46Z"} 2024-03-04T17:52:46.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:46.769 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:46.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:46.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.885240_641.1 2024-03-04T17:52:51.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:51.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:51.771 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:51.816 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:51.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:51.883 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:51.884+0000 7f4337e22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:51.884 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:51.884+0000 7f4337e22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:51.909 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:51Z"} 2024-03-04T17:52:51.909 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:51Z"} 2024-03-04T17:52:51.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:51.910 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:51.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:51.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:52:56.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:56.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:52:56.911 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:52:56.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:52:56.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:52:57.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:57.026+0000 7f5c1493c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:57.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:52:57.027+0000 7f5c1493c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:52:57.051 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:57Z"} 2024-03-04T17:52:57.051 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:57Z"} 2024-03-04T17:52:57.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:52:57.052 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:57.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:52:57.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:53:02.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:02.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:02.053 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:02.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:02.100 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:02.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:02.159+0000 7f288f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:02.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:02.160+0000 7f288f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:02.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:02Z"} 2024-03-04T17:53:02.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:02Z"} 2024-03-04T17:53:02.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:02.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:02.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:02.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:53:07.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:07.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:07.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:07.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:07.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:07.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:07.285+0000 7fb53243a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:07.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:07.286+0000 7fb53243a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:07.308 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:07Z"} 2024-03-04T17:53:07.309 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:07Z"} 2024-03-04T17:53:07.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:07.309 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:07.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:07.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:53:12.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:12.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:12.310 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:12.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:12.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:12.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:12.415+0000 7fae27339800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:12.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:12.415+0000 7fae27339800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:12.437 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:12Z"} 2024-03-04T17:53:12.437 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:12Z"} 2024-03-04T17:53:12.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:12.438 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:12.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:12.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:53:17.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:17.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:17.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:17.487 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:17.487 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:17.548 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:17.549+0000 7fac0072a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:17.549 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:17.549+0000 7fac0072a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:17.583 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:17Z"} 2024-03-04T17:53:17.583 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:17Z"} 2024-03-04T17:53:17.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:17.583 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:17.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:17.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:53:22.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:22.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:22.585 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:22.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:22.692+0000 7fdb1ea9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:22.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:22.693+0000 7fdb1ea9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:22.722 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:22Z"} 2024-03-04T17:53:22.723 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:22Z"} 2024-03-04T17:53:22.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:22.723 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:22.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:22.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:53:27.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:27.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:27.725 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:27.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:27.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:27.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:27.845+0000 7fd2e2c34800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:27.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:27.846+0000 7fd2e2c34800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:27.869 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:27Z"} 2024-03-04T17:53:27.869 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:27Z"} 2024-03-04T17:53:27.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:27.870 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:27.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:27.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:53:32.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:32.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:32.871 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:32.920 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:32.920 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:32.979 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:32.979+0000 7f392bb22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:32.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:32.980+0000 7f392bb22800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:33.015 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:32Z"} 2024-03-04T17:53:33.015 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:32Z"} 2024-03-04T17:53:33.015 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:33.015 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:33.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:33.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:53:38.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:38.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:38.017 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:38.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:38.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:38.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:38.115+0000 7f282252f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:38.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:38.116+0000 7f282252f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:38.138 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:38Z"} 2024-03-04T17:53:38.138 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:38Z"} 2024-03-04T17:53:38.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:38.139 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:38.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:38.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:53:43.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:43.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:43.141 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:43.236 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:43.236+0000 7ff1bca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:43.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:43.237+0000 7ff1bca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:43.266 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:43Z"} 2024-03-04T17:53:43.266 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:43Z"} 2024-03-04T17:53:43.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:43.267 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:43.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:43.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:53:48.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:48.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:48.269 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:48.317 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:48.317 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:48.387+0000 7ffb67130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:48.388+0000 7ffb67130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:48.416 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:48Z"} 2024-03-04T17:53:48.417 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:48Z"} 2024-03-04T17:53:48.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:48.417 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:48.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:48.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:53:53.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:53.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:53.419 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:53.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:53.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:53.537 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:53.537+0000 7f6a59813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:53.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:53.538+0000 7f6a59813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:53.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:53Z"} 2024-03-04T17:53:53.565 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:53Z"} 2024-03-04T17:53:53.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:53.565 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:53.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:53.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:53:58.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:58.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:53:58.567 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:53:58.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:53:58.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:53:58.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:58.685+0000 7ffaa6830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.686 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:53:58.686+0000 7ffaa6830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:53:58.715 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T17:53:58.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:53:58.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:58.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:53:58.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:54:03.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:03.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:03.718 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:03.771 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:03.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:03.824 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:03.826+0000 7f415b330800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:03.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:03.826+0000 7f415b330800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:03.849 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:03Z"} 2024-03-04T17:54:03.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:03Z"} 2024-03-04T17:54:03.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:03.850 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:03.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:03.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:54:08.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:08.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:08.852 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:08.932 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:08.932 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:09.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:09.053+0000 7f8720e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:09.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:09.054+0000 7f8720e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:09.080 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:09Z"} 2024-03-04T17:54:09.080 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:09Z"} 2024-03-04T17:54:09.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:09.081 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:09.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:09.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:54:14.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:14.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:14.083 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:14.164 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:14.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:14.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:14.277+0000 7f9983a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:14.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:14.278+0000 7f9983a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:14.302 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:14Z"} 2024-03-04T17:54:14.302 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:14Z"} 2024-03-04T17:54:14.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:14.303 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:14.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:14.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:54:19.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:19.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:19.305 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:19.358 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:19.358 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:19.427 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:19.427+0000 7fceddb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:19.428 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:19.428+0000 7fceddb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:19.452 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:19Z"} 2024-03-04T17:54:19.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:19Z"} 2024-03-04T17:54:19.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:19.453 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:19.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:19.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T17:54:24.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:24.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:24.455 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:24.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:24.564+0000 7f0f8761e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:24.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:24.564+0000 7f0f8761e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:24.589 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:24Z"} 2024-03-04T17:54:24.589 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:24Z"} 2024-03-04T17:54:24.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:24.590 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:24.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:24.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:54:29.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:29.591 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:29.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:29.698+0000 7fbedec10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:29.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:29.699+0000 7fbedec10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:29Z"} 2024-03-04T17:54:29.721 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:29Z"} 2024-03-04T17:54:29.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:29.721 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:29.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:29.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T17:54:34.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:34.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:34.723 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:34.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:34.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:34.831 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:34.831+0000 7f9b86509800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:34.832 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:34.832+0000 7f9b86509800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:34.855 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:34Z"} 2024-03-04T17:54:34.855 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:34Z"} 2024-03-04T17:54:34.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:34.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:34.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:34.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:54:39.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:39.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:39.857 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:39.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:39.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:39.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:39.959+0000 7f2b02b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:39.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:39.960+0000 7f2b02b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:39.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:39Z"} 2024-03-04T17:54:39.987 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:39Z"} 2024-03-04T17:54:39.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:39.987 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:39.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:39.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:54:44.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:44.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:44.989 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:45.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:45.036 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:45.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:45.095+0000 7feba743c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:45.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:45.096+0000 7feba743c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:45.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:45Z"} 2024-03-04T17:54:45.126 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:45Z"} 2024-03-04T17:54:45.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:45.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:45.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:45.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:54:50.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:50.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:50.128 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:50.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:50.177 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:50.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:50.229+0000 7f704fac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:50.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:50.230+0000 7f704fac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:50.255 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:50Z"} 2024-03-04T17:54:50.255 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:50Z"} 2024-03-04T17:54:50.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:50.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:50.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:50.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.890340_643.1 2024-03-04T17:54:55.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:55.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:54:55.257 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:54:55.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:54:55.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:54:55.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:55.357+0000 7fc966830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:55.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:54:55.358+0000 7fc966830800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:54:55.380 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:55Z"} 2024-03-04T17:54:55.380 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:55Z"} 2024-03-04T17:54:55.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:54:55.381 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:55.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:54:55.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.828757_695.1 2024-03-04T17:55:00.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:00.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:00.382 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:00.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:00.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:00.480+0000 7f705208c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:00.481+0000 7f705208c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:00.504 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:00Z"} 2024-03-04T17:55:00.504 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:00Z"} 2024-03-04T17:55:00.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:00.505 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:00.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:00.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T17:55:05.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:05.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:05.506 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:05.556 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:05.557 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:05.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:05.622+0000 7f8c2e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:05.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:05.623+0000 7f8c2e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:05.650 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:05Z"} 2024-03-04T17:55:05.650 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:05Z"} 2024-03-04T17:55:05.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:05.651 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:05.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:05.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T17:55:10.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:10.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:10.652 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:10.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:10.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:10.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:10.763+0000 7efdf242e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:10.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:10.764+0000 7efdf242e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:10.791 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:10Z"} 2024-03-04T17:55:10.791 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:10Z"} 2024-03-04T17:55:10.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:10.792 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:10.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:10.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:55:15.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:15.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:15.793 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:15.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:15.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:15.891 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:15.891+0000 7f38a652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:15.892 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:15.892+0000 7f38a652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:15.914 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:15Z"} 2024-03-04T17:55:15.914 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:15Z"} 2024-03-04T17:55:15.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:15.915 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:15.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:15.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.125073_710.1 2024-03-04T17:55:20.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:20.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:20.916 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:21.018+0000 7f965af2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:21.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:21.019+0000 7f965af2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:21.049 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:21Z"} 2024-03-04T17:55:21.049 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:21Z"} 2024-03-04T17:55:21.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:21.050 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:21.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:21.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:55:26.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:26.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:26.052 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:26.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:26.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:26.160 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:26.160+0000 7f1c74728800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:26.161 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:26.161+0000 7f1c74728800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:26.186 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:26Z"} 2024-03-04T17:55:26.187 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:26Z"} 2024-03-04T17:55:26.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:26.187 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:26.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:26.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T17:55:31.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:31.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:31.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:31.238 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:31.238 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:31.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:31.302+0000 7fcd42309800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:31.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:31.303+0000 7fcd42309800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:31.329 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:31Z"} 2024-03-04T17:55:31.330 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:31Z"} 2024-03-04T17:55:31.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:31.330 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:31.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:31.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:55:36.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:36.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:36.332 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:36.396 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:36.396 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:36.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:36.463+0000 7f3d30542800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:36.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:36.464+0000 7f3d30542800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:36.493 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:36Z"} 2024-03-04T17:55:36.493 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:36Z"} 2024-03-04T17:55:36.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:36.493 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:36.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:36.493 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.689443_569.1 2024-03-04T17:55:41.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:41.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:41.495 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:41.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:41.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:41.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:41.634+0000 7f488a31f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:41.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:41.637+0000 7f488a31f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:41.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:41Z"} 2024-03-04T17:55:41.727 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:41Z"} 2024-03-04T17:55:41.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:41.727 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:41.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:41.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:55:46.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:46.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:46.729 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:46.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:46.910+0000 7f6121f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:46.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:46.911+0000 7f6121f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:46.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:46Z"} 2024-03-04T17:55:46.997 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:46Z"} 2024-03-04T17:55:46.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:46.997 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:46.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:46.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:55:51.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:51.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:52.000 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:52.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:52.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:52.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:52.136+0000 7fdc3f52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:52.136 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:52.137+0000 7fdc3f52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:52.164 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:52Z"} 2024-03-04T17:55:52.164 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:52Z"} 2024-03-04T17:55:52.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:52.165 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:52.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:52.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:55:57.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:57.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:55:57.167 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:55:57.212 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:55:57.213 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:55:57.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:57.270+0000 7fc9ff531800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:57.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:55:57.271+0000 7fc9ff531800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:55:57.296 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:57Z"} 2024-03-04T17:55:57.297 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:57Z"} 2024-03-04T17:55:57.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:55:57.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:57.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:55:57.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T17:56:02.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:02.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:02.300 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:02.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:02.347 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:02.407 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:02.408+0000 7f215ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:02.408 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:02.408+0000 7f215ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:02.440 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:02Z"} 2024-03-04T17:56:02.441 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:02Z"} 2024-03-04T17:56:02.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:02.441 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:02.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:02.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.689443_569.1 2024-03-04T17:56:07.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:07.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:07.442 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:07.488 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:07.488 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:07.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:07.538+0000 7febbc410800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:07.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:07.539+0000 7febbc410800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:07.565 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:07Z"} 2024-03-04T17:56:07.566 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:07Z"} 2024-03-04T17:56:07.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:07.566 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:07.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:07.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T17:56:12.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:12.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:12.567 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:12.681+0000 7f40e3a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:12.682+0000 7f40e3a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:12.714 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:12Z"} 2024-03-04T17:56:12.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:12Z"} 2024-03-04T17:56:12.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:12.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:12.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:12.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T17:56:17.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:17.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:17.717 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:17.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:17.766 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:17.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:17.817+0000 7f5f0ab31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:17.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:17.818+0000 7f5f0ab31800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:17.844 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:17Z"} 2024-03-04T17:56:17.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:17Z"} 2024-03-04T17:56:17.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:17.845 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:17.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:17.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:56:22.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:22.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:22.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:22.893 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:22.894 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:22.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:22.962+0000 7fed75f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:22.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:22.962+0000 7fed75f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:22.990 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:22Z"} 2024-03-04T17:56:22.990 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:22Z"} 2024-03-04T17:56:22.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:22.990 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:22.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:22.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:56:27.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:27.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:27.992 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:28.038 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:28.038 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:28.087 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:28.087+0000 7f57f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:28.088 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:28.088+0000 7f57f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:28Z"} 2024-03-04T17:56:28.116 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:28Z"} 2024-03-04T17:56:28.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:28.117 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:28.117 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:28.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:56:33.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:33.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:33.118 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:33.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:33.224+0000 7f2d82c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:33.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:33.225+0000 7f2d82c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:33.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:33Z"} 2024-03-04T17:56:33.248 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:33Z"} 2024-03-04T17:56:33.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:33.248 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:33.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:33.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T17:56:38.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:38.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:38.250 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:38.356 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:38.356+0000 7faf39537800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:38.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:38.357+0000 7faf39537800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:38.379 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:38Z"} 2024-03-04T17:56:38.379 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:38Z"} 2024-03-04T17:56:38.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:38.380 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:38.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:38.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T17:56:43.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:43.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:43.382 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:43.435 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:43.435 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:43.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:43.561+0000 7fc1eb038800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:43.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:43.571+0000 7fc1eb038800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:43.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:43Z"} 2024-03-04T17:56:43.646 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:43Z"} 2024-03-04T17:56:43.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:43.646 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:43.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:43.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:56:48.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:48.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:48.648 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:48.697 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:48.697 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:48.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:48.748+0000 7fe471424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:48.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:48.749+0000 7fe471424800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:48.776 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:48Z"} 2024-03-04T17:56:48.776 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:48Z"} 2024-03-04T17:56:48.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:48.777 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:48.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:48.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T17:56:53.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:53.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:53.779 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:53.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:53.825 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:53.874+0000 7f6e3d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:53.875 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:53.875+0000 7f6e3d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:53.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:53Z"} 2024-03-04T17:56:53.902 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:53Z"} 2024-03-04T17:56:53.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:53.903 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:53.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:53.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.684679_567.1 2024-03-04T17:56:58.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:58.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:56:58.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:56:58.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:56:58.952 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:56:59.001 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:59.001+0000 7f171ab2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:59.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:56:59.002+0000 7f171ab2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:56:59.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:59Z"} 2024-03-04T17:56:59.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:59Z"} 2024-03-04T17:56:59.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:56:59.034 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:59.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:56:59.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:57:04.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:04.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:04.035 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:04.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:04.081 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:04.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:04.142+0000 7f1deba1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:04.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:04.143+0000 7f1deba1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:04.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:04Z"} 2024-03-04T17:57:04.165 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:04Z"} 2024-03-04T17:57:04.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:04.166 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:04.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:04.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T17:57:09.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:09.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:09.167 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:09.213 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:09.213 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:09.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:09.271+0000 7f5df1a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:09.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:09.271+0000 7f5df1a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:09.294 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:09Z"} 2024-03-04T17:57:09.294 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:09Z"} 2024-03-04T17:57:09.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:09.295 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:09.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:09.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.616944_562.1 2024-03-04T17:57:14.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:14.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:14.296 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:14.343 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:14.343 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:14.395 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:14.395+0000 7f9220a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:14.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:14.396+0000 7f9220a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:14.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:14Z"} 2024-03-04T17:57:14.419 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:14Z"} 2024-03-04T17:57:14.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:14.419 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:14.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:14.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T17:57:19.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:19.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:19.422 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:19.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:19.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:19.515 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:19.515+0000 7fa85393c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:19.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:19.516+0000 7fa85393c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:19.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:19Z"} 2024-03-04T17:57:19.539 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:19Z"} 2024-03-04T17:57:19.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:19.539 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:19.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:19.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T17:57:24.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:24.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:24.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:24.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:24.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:24.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:24.648+0000 7f99993af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:24.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:24.649+0000 7f99993af800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:24.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:24Z"} 2024-03-04T17:57:24.677 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:24Z"} 2024-03-04T17:57:24.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:24.677 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:24.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:24.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.684679_567.1 2024-03-04T17:57:29.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:29.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:29.679 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:29.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:29.727 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:29.789 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:29.789+0000 7f8acd330800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:29.790 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:29.790+0000 7f8acd330800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:29Z"} 2024-03-04T17:57:29.820 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:29Z"} 2024-03-04T17:57:29.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:29.821 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:29.821 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:29.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T17:57:34.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:34.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:34.823 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:34.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:34.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:34.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:34.939+0000 7fc69f611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:34.940 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:34.940+0000 7fc69f611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:34.965 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:34Z"} 2024-03-04T17:57:34.966 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:34Z"} 2024-03-04T17:57:34.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:34.966 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:34.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:34.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T17:57:39.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:39.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:39.967 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:40.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:40.077+0000 7f57d3930800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:40.078 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:40.078+0000 7f57d3930800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:40.110 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:40Z"} 2024-03-04T17:57:40.110 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:40Z"} 2024-03-04T17:57:40.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:40.110 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:40.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:40.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.154969_713.1 2024-03-04T17:57:45.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:45.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:45.113 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:45.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:45.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:45.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:45.216+0000 7f0427024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:45.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:45.217+0000 7f0427024800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:45.250 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:45Z"} 2024-03-04T17:57:45.250 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:45Z"} 2024-03-04T17:57:45.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:45.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:45.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:45.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:57:50.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:50.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:50.252 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:50.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:50.374+0000 7f0669611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:50.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:50.375+0000 7f0669611800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:50.402 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:50Z"} 2024-03-04T17:57:50.402 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:50Z"} 2024-03-04T17:57:50.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:50.403 DEBUG:rgw_multi.tests:log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:50.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:50.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T17:57:55.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2024-03-04T17:57:55.407 DEBUG:boto:path=/sbnvju-72/ 2024-03-04T17:57:55.407 DEBUG:boto:auth_path=/sbnvju-72/ 2024-03-04T17:57:55.407 DEBUG:boto:Method: PUT 2024-03-04T17:57:55.407 DEBUG:boto:Path: /sbnvju-72/ 2024-03-04T17:57:55.407 DEBUG:boto:Data: 2024-03-04T17:57:55.407 DEBUG:boto:Headers: {} 2024-03-04T17:57:55.408 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T17:57:55.408 DEBUG:boto:Port: 8000 2024-03-04T17:57:55.408 DEBUG:boto:Params: {} 2024-03-04T17:57:55.408 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T17:57:55.408 DEBUG:boto:Token: None 2024-03-04T17:57:55.409 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 17:57:55 GMT /sbnvju-72/ 2024-03-04T17:57:55.409 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:XhcLW6MqqyHr8lloCt8aEigoMLk= 2024-03-04T17:57:55.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhcLW6MqqyHr8lloCt8aEigoMLk=', 'Content-Length': '0'} 2024-03-04T17:57:55.435 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c1839f504aa84e1-0065e60ba3-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T17:57:55.435 DEBUG:rgw_multi.tests:created bucket=sbnvju-72 2024-03-04T17:57:55.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:55.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:55.436 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T17:57:55.562 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709575075.434129_1008.1","last_update":"2024-03-04T17:57:55.434129Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T17:57:55.562 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:55.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T17:57:55.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:55.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:57:55.563 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:57:55.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:57:55.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:57:55.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:55.760+0000 7f66f3f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:55.761 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:57:55.761+0000 7f66f3f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:57:55.795 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:55Z"} 2024-03-04T17:57:55.795 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:55Z"} 2024-03-04T17:57:55.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:57:55.795 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:55.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:57:55.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.608330_559.1 2024-03-04T17:58:00.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:00.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:00.797 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:00.859 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:00.859 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:00.915 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:00.915+0000 7f139f12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:00.916 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:00.916+0000 7f139f12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:00.949 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:00Z"} 2024-03-04T17:58:00.949 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:00Z"} 2024-03-04T17:58:00.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:00.950 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:00.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:00.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T17:58:05.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:05.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:05.951 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:05.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:05.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:06.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:06.062+0000 7fcea401b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:06.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:06.062+0000 7fcea401b800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:06.090 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:06Z"} 2024-03-04T17:58:06.090 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:06Z"} 2024-03-04T17:58:06.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:06.091 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:06.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:06.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T17:58:11.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:11.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:11.093 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:11.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:11.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:11.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:11.205+0000 7f836992d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:11.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:11.206+0000 7f836992d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:11.229 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:11Z"} 2024-03-04T17:58:11.229 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:11Z"} 2024-03-04T17:58:11.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:11.229 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:11.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:11.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T17:58:16.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:16.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:16.231 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:16.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:16.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:16.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:16.327+0000 7f0677c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:16.327+0000 7f0677c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:16.349 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:16Z"} 2024-03-04T17:58:16.349 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:16Z"} 2024-03-04T17:58:16.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:16.350 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:16.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:16.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T17:58:21.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:21.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:21.351 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:21.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:21.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:21.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:21.470+0000 7f12d5c10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:21.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:21.470+0000 7f12d5c10800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:21.503 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:21Z"} 2024-03-04T17:58:21.503 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:21Z"} 2024-03-04T17:58:21.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:21.504 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:21.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:21.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.688202_568.1 2024-03-04T17:58:26.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:26.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:26.506 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:26.553 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:26.553 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:26.601 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:26.601+0000 7f08e6524800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:26.601 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:26.601+0000 7f08e6524800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:26Z"} 2024-03-04T17:58:26.632 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:26Z"} 2024-03-04T17:58:26.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:26.632 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:26.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:26.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T17:58:31.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:31.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:31.635 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:31.682 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:31.682 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:31.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:31.734+0000 7f63f3642800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:31.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:31.735+0000 7f63f3642800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:31.760 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:31Z"} 2024-03-04T17:58:31.760 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:31Z"} 2024-03-04T17:58:31.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:31.761 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:31.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:31.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T17:58:36.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:36.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:36.762 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:36.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:36.874+0000 7f82c6c38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:36.875 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:36.875+0000 7f82c6c38800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:36.899 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:36Z"} 2024-03-04T17:58:36.899 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:36Z"} 2024-03-04T17:58:36.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:36.899 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:36.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:36.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T17:58:41.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:41.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:41.900 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:41.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:41.953 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:42.017+0000 7fcbb8c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:42.018+0000 7fcbb8c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:42.044 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:42Z"} 2024-03-04T17:58:42.044 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:42Z"} 2024-03-04T17:58:42.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:42.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:42.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:42.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T17:58:47.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:47.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:47.046 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:47.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:47.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:47.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:47.152+0000 7fb534e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:47.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:47.153+0000 7fb534e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:47.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:47Z"} 2024-03-04T17:58:47.183 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:47Z"} 2024-03-04T17:58:47.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:47.184 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:47.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:47.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.637258_563.1 2024-03-04T17:58:52.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:52.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:52.186 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:52.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:52.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:52.407+0000 7f92777a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:52.409 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:52.408+0000 7f92777a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:52.444 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:52Z"} 2024-03-04T17:58:52.445 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:52Z"} 2024-03-04T17:58:52.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:52.445 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:52.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:52.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T17:58:57.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:57.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:58:57.447 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:58:57.495 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:58:57.495 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:58:57.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:57.565+0000 7f8373709800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:57.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:58:57.566+0000 7f8373709800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:58:57.593 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:57Z"} 2024-03-04T17:58:57.593 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:57Z"} 2024-03-04T17:58:57.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:58:57.594 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:57.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:58:57.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T17:59:02.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:02.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:02.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:02.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:02.642 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:02.705 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:02.705+0000 7fdb4e92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:02.706 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:02.706+0000 7fdb4e92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:02.736 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:02Z"} 2024-03-04T17:59:02.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:02Z"} 2024-03-04T17:59:02.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:02.737 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:02.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:02.737 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T17:59:07.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:07.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:07.738 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:07.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:07.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:07.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:07.841+0000 7f711388c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:07.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:07.842+0000 7f711388c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:07.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:07Z"} 2024-03-04T17:59:07.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:07Z"} 2024-03-04T17:59:07.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:07.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:07.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:07.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T17:59:12.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:12.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:12.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:12.914 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:12.914 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:12.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:12.965+0000 7fab5c813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:12.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:12.966+0000 7fab5c813800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:12.989 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:12Z"} 2024-03-04T17:59:12.989 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:12Z"} 2024-03-04T17:59:12.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:12.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:12.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:12.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.121915_709.1 2024-03-04T17:59:17.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:17.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:17.992 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:18.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:18.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:18.102 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:18.102+0000 7fb035617800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:18.103 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:18.103+0000 7fb035617800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:18.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:18Z"} 2024-03-04T17:59:18.126 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:18Z"} 2024-03-04T17:59:18.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:18.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:18.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:18.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T17:59:23.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:23.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:23.128 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:23.175 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:23.175 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:23.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:23.239+0000 7f536162e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:23.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:23.239+0000 7f536162e800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:23.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:23Z"} 2024-03-04T17:59:23.264 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:23Z"} 2024-03-04T17:59:23.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:23.264 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:23.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:23.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.996460_704.1 2024-03-04T17:59:28.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:28.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:28.266 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:28.315 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:28.315 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:28.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:28.390+0000 7f8d9e444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:28.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:28.391+0000 7f8d9e444800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:28.422 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:28Z"} 2024-03-04T17:59:28.423 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:28Z"} 2024-03-04T17:59:28.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:28.423 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:28.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:28.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T17:59:33.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:33.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:33.424 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:33.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:33.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:33.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:33.527+0000 7f4e8880f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:33.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:33.528+0000 7f4e8880f800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:33.552 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:33Z"} 2024-03-04T17:59:33.552 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:33Z"} 2024-03-04T17:59:33.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:33.552 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:33.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:33.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.688202_568.1 2024-03-04T17:59:38.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:38.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:38.555 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:38.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:38.603 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:38.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:38.669+0000 7f0d32e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:38.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:38.670+0000 7f0d32e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:38.696 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:38Z"} 2024-03-04T17:59:38.697 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:38Z"} 2024-03-04T17:59:38.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:38.697 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:38.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:38.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T17:59:43.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:43.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:43.699 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:43.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:43.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:43.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:43.815+0000 7f3b31130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:43.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:43.816+0000 7f3b31130800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:43.842 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:43Z"} 2024-03-04T17:59:43.842 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:43Z"} 2024-03-04T17:59:43.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:43.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:43.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:43.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.608330_559.1 2024-03-04T17:59:48.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:48.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:48.844 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:48.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:48.892 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:48.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:48.953+0000 7f6c2ec8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:48.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:48.953+0000 7f6c2ec8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:48.987 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:48Z"} 2024-03-04T17:59:48.987 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:48Z"} 2024-03-04T17:59:48.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:48.988 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:48.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:48.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T17:59:53.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:53.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:53.989 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:54.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:54.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:54.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:54.114+0000 7f43bcb37800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:54.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:54.115+0000 7f43bcb37800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:54.142 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:54Z"} 2024-03-04T17:59:54.142 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:54Z"} 2024-03-04T17:59:54.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:54.142 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:54.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:54.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T17:59:59.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:59.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T17:59:59.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T17:59:59.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T17:59:59.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T17:59:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:59.250+0000 7fcc55539800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:59.251 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T17:59:59.251+0000 7fcc55539800 20 RGW-SYNC:meta: read sync status 2024-03-04T17:59:59.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:59Z"} 2024-03-04T17:59:59.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:59Z"} 2024-03-04T17:59:59.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T17:59:59.274 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:59.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T17:59:59.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T18:00:04.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:04.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:04.276 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:04.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:04.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:04.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:04.443+0000 7f802601b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:04.447 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:04.447+0000 7f802601b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:04.522 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:04Z"} 2024-03-04T18:00:04.523 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:04Z"} 2024-03-04T18:00:04.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:04.523 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:04.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:04.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T18:00:09.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:09.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:09.525 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:09.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:09.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:09.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:09.689+0000 7fa9d5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:09.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:09.690+0000 7fa9d5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:09.713 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:09Z"} 2024-03-04T18:00:09.714 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:09Z"} 2024-03-04T18:00:09.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:09.714 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:09.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:09.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.838159_698.1 2024-03-04T18:00:14.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:14.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:14.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:14.763 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:14.763 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:14.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:14.822+0000 7fc78e013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:14.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:14.823+0000 7fc78e013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:14.845 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:14Z"} 2024-03-04T18:00:14.846 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:14Z"} 2024-03-04T18:00:14.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:14.846 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:14.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:14.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T18:00:19.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:19.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:19.848 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:19.896 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:19.897 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:19.946 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:19.946+0000 7f0197509800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:19.947+0000 7f0197509800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:19.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:19Z"} 2024-03-04T18:00:19.971 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:19Z"} 2024-03-04T18:00:19.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:19.971 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:19.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:19.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T18:00:24.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:24.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:24.973 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:25.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:25.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:25.080 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:25.081+0000 7fa0cab2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:25.081 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:25.082+0000 7fa0cab2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:25.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:25Z"} 2024-03-04T18:00:25.113 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:25Z"} 2024-03-04T18:00:25.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:25.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:25.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:25.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:00:30.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:30.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:30.115 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:30.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:30.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:30.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:30.225+0000 7f9fee2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:30.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:30.226+0000 7f9fee2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:30.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:30Z"} 2024-03-04T18:00:30.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:30Z"} 2024-03-04T18:00:30.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:30.249 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:30.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:30.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T18:00:35.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:35.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:35.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:35.348 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:35.348+0000 7f39e1432800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:35.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:35.349+0000 7f39e1432800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:35.372 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:35Z"} 2024-03-04T18:00:35.372 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:35Z"} 2024-03-04T18:00:35.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:35.373 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:35.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:35.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T18:00:40.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:40.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:40.374 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:40.421 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:40.421 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:40.475 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:40.475+0000 7fa5b6839800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:40.476 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:40.476+0000 7fa5b6839800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:40.500 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:40Z"} 2024-03-04T18:00:40.500 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:40Z"} 2024-03-04T18:00:40.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:40.501 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:40.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:40.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T18:00:45.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:45.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:45.502 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:45.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:45.622+0000 7f2017c67800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:45.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:45.623+0000 7f2017c67800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:45.650 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:45Z"} 2024-03-04T18:00:45.650 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:45Z"} 2024-03-04T18:00:45.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:45.651 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:45.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:45.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:00:50.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:50.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:50.652 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:50.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:50.760+0000 7f28aaa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:50.761 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:50.761+0000 7f28aaa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:50.783 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:50Z"} 2024-03-04T18:00:50.783 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:50Z"} 2024-03-04T18:00:50.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:50.784 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:50.784 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:50.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.002120_706.1 2024-03-04T18:00:55.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:55.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:00:55.785 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:00:55.835 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:00:55.835 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:00:55.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:55.897+0000 7feba9e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:55.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:00:55.898+0000 7feba9e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:00:55.925 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:55Z"} 2024-03-04T18:00:55.925 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:55Z"} 2024-03-04T18:00:55.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:00:55.926 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:55.926 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:00:55.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T18:01:00.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:00.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:00.927 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:00.975 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:00.976 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:01.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:01.031+0000 7fbabc612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:01.032 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:01.032+0000 7fbabc612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:01.054 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:01Z"} 2024-03-04T18:01:01.054 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:01Z"} 2024-03-04T18:01:01.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:01.055 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:01.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:01.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T18:01:06.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:06.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:06.056 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:06.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:06.155+0000 7f8db6f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:06.156+0000 7f8db6f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:06.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:06Z"} 2024-03-04T18:01:06.180 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:06Z"} 2024-03-04T18:01:06.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:06.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:06.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:06.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T18:01:11.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:11.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:11.182 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:11.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:11.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:11.293+0000 7f5398a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:11.294+0000 7f5398a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:11.327 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:11Z"} 2024-03-04T18:01:11.327 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:11Z"} 2024-03-04T18:01:11.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:11.328 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:11.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:11.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T18:01:16.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:16.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:16.329 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:16.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:16.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:16.447 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:16.447+0000 7f2a85030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:16.448 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:16.448+0000 7f2a85030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:16.471 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:16Z"} 2024-03-04T18:01:16.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:16Z"} 2024-03-04T18:01:16.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:16.471 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:16.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:16.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:01:21.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:21.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:21.473 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:21.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:21.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:21.579 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:21.580+0000 7f8d4a00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:21.580 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:21.581+0000 7f8d4a00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:21.605 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:21Z"} 2024-03-04T18:01:21.606 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:21Z"} 2024-03-04T18:01:21.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:21.606 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:21.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:21.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T18:01:26.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:26.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:26.608 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:26.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:26.661 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:26.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:26.733+0000 7f4ae0d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:26.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:26.734+0000 7f4ae0d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:26.773 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:26Z"} 2024-03-04T18:01:26.773 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:26Z"} 2024-03-04T18:01:26.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:26.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:26.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:26.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T18:01:31.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:31.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:31.775 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:31.853 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:31.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:31.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:31.931+0000 7f2d6513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:31.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:31.932+0000 7f2d6513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:31.960 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:31Z"} 2024-03-04T18:01:31.960 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:31Z"} 2024-03-04T18:01:31.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:31.961 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:31.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:31.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:01:36.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:36.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:36.962 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:37.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:37.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:37.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:37.061+0000 7fc0e9131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:37.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:37.062+0000 7fc0e9131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:37.087 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:37Z"} 2024-03-04T18:01:37.087 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:37Z"} 2024-03-04T18:01:37.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:37.088 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:37.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:37.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T18:01:42.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:42.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:42.090 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:42.137 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:42.137 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:42.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:42.203+0000 7fb114d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:42.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:42.204+0000 7fb114d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:42.232 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:42Z"} 2024-03-04T18:01:42.232 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:42Z"} 2024-03-04T18:01:42.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:42.233 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:42.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:42.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T18:01:47.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:47.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:47.234 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:47.281 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:47.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:47.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:47.350+0000 7fe8b132d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:47.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:47.351+0000 7fe8b132d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:47.383 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:47Z"} 2024-03-04T18:01:47.383 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:47Z"} 2024-03-04T18:01:47.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:47.384 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:47.384 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:47.384 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T18:01:52.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:52.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:52.385 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:52.433 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:52.433 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:52.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:52.495+0000 7f773ff3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:52.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:52.496+0000 7f773ff3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:52.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:52Z"} 2024-03-04T18:01:52.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:52Z"} 2024-03-04T18:01:52.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:52.530 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:52.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:52.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T18:01:57.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:57.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:01:57.532 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:01:57.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:01:57.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:01:57.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:57.637+0000 7f902a12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:57.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:01:57.638+0000 7f902a12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:01:57.660 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:57Z"} 2024-03-04T18:01:57.660 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:57Z"} 2024-03-04T18:01:57.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:01:57.661 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:57.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:01:57.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.002120_706.1 2024-03-04T18:02:02.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:02.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:02.662 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:02.710 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:02.710 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:02.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:02.766+0000 7f70a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:02.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:02.767+0000 7f70a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:02.795 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:02Z"} 2024-03-04T18:02:02.795 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:02Z"} 2024-03-04T18:02:02.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:02.795 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:02.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:02.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.116657_707.1 2024-03-04T18:02:07.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:07.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:07.797 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:07.845 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:07.921 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:07.921+0000 7f76ae931800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:07.921 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:07.921+0000 7f76ae931800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:07.955 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:07Z"} 2024-03-04T18:02:07.955 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:07Z"} 2024-03-04T18:02:07.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:07.955 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:07.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:07.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.641846_565.1 2024-03-04T18:02:12.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:12.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:12.957 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:13.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:13.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:13.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:13.061+0000 7f6198a2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:13.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:13.062+0000 7f6198a2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:13.083 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:13Z"} 2024-03-04T18:02:13.084 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:13Z"} 2024-03-04T18:02:13.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:13.084 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:13.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:13.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T18:02:18.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:18.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:18.086 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:18.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:18.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:18.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:18.203+0000 7f58c2f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:18.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:18.204+0000 7f58c2f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:18.239 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:18Z"} 2024-03-04T18:02:18.239 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:18Z"} 2024-03-04T18:02:18.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:18.239 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:18.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:18.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T18:02:23.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:23.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:23.241 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:23.291 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:23.291 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:23.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:23.360+0000 7f12f7f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:23.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:23.361+0000 7f12f7f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:23.385 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:23Z"} 2024-03-04T18:02:23.385 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:23Z"} 2024-03-04T18:02:23.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:23.386 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:23.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:23.386 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T18:02:28.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:28.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:28.387 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:28.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:28.438 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:28.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:28.508+0000 7f6d2b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:28.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:28.509+0000 7f6d2b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:28.532 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:28Z"} 2024-03-04T18:02:28.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:28Z"} 2024-03-04T18:02:28.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:28.532 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:28.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:28.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T18:02:33.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:33.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:33.534 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:33.587 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:33.587 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:33.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:33.667+0000 7fc64c142800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:33.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:33.668+0000 7fc64c142800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:33.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:33Z"} 2024-03-04T18:02:33.702 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:33Z"} 2024-03-04T18:02:33.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:33.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:33.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:33.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.987908_703.1 2024-03-04T18:02:38.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:38.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:38.704 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:38.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:38.758 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:38.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:38.822+0000 7fa0b17a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:38.824 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:38.823+0000 7fa0b17a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:38.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:38Z"} 2024-03-04T18:02:38.848 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:38Z"} 2024-03-04T18:02:38.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:38.848 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:38.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:38.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:02:43.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:43.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:43.849 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:43.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:43.905 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:43.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:43.965+0000 7f67cc430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:43.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:43.966+0000 7f67cc430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:44.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:43Z"} 2024-03-04T18:02:44.012 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:43Z"} 2024-03-04T18:02:44.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:44.012 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:44.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:44.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T18:02:49.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:49.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:49.013 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:49.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:49.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:49.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:49.213+0000 7fdd45b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:49.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:49.213+0000 7fdd45b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:49.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:49Z"} 2024-03-04T18:02:49.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:49Z"} 2024-03-04T18:02:49.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:49.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:49.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:49.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T18:02:54.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:54.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:54.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:54.321 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:54.322 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:54.378 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:54.378+0000 7f0408224800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:54.379 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:54.379+0000 7f0408224800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:54.402 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:54Z"} 2024-03-04T18:02:54.402 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:54Z"} 2024-03-04T18:02:54.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:54.403 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:54.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:54.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T18:02:59.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:59.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:02:59.405 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:02:59.453 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:02:59.454 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:02:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:59.516+0000 7fbb49e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:59.517 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:02:59.516+0000 7fbb49e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:02:59.539 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:59Z"} 2024-03-04T18:02:59.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:59Z"} 2024-03-04T18:02:59.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:02:59.540 DEBUG:rgw_multi.tests:log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:59.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:02:59.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.641846_565.1 2024-03-04T18:03:04.544 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2024-03-04T18:03:04.545 DEBUG:boto:path=/sbnvju-73/ 2024-03-04T18:03:04.546 DEBUG:boto:auth_path=/sbnvju-73/ 2024-03-04T18:03:04.546 DEBUG:boto:Method: PUT 2024-03-04T18:03:04.546 DEBUG:boto:Path: /sbnvju-73/ 2024-03-04T18:03:04.546 DEBUG:boto:Data: 2024-03-04T18:03:04.546 DEBUG:boto:Headers: {} 2024-03-04T18:03:04.546 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:03:04.546 DEBUG:boto:Port: 8000 2024-03-04T18:03:04.546 DEBUG:boto:Params: {} 2024-03-04T18:03:04.547 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:03:04.547 DEBUG:boto:Token: None 2024-03-04T18:03:04.548 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:03:04 GMT /sbnvju-73/ 2024-03-04T18:03:04.548 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:+YLaEyMoacgfCSgheKofs4XRd1Y= 2024-03-04T18:03:04.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+YLaEyMoacgfCSgheKofs4XRd1Y=', 'Content-Length': '0'} 2024-03-04T18:03:04.573 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e3b541d3932e0b1-0065e60cd8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:03:04.574 DEBUG:rgw_multi.tests:created bucket=sbnvju-73 2024-03-04T18:03:04.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:04.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:04.574 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:03:04.698 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T18:03:04.699 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:04.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:03:04.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:04.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:04.699 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:04.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:04.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:04.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:04.896+0000 7f2788a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:04.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:04.898+0000 7f2788a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:04.947 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:04Z"} 2024-03-04T18:03:04.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:04Z"} 2024-03-04T18:03:04.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:04.948 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:04.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:04.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:03:09.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:09.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:09.950 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:10.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:10.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:10.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:10.062+0000 7fb08d52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:10.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:10.063+0000 7fb08d52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:10.086 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:10Z"} 2024-03-04T18:03:10.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:10Z"} 2024-03-04T18:03:10.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:10.087 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:10.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:10.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:03:15.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:15.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:15.089 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:15.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:15.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:15.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:15.182+0000 7fd54111f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:15.183 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:15.183+0000 7fd54111f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:15.207 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:15Z"} 2024-03-04T18:03:15.207 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:15Z"} 2024-03-04T18:03:15.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:15.208 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:15.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:15.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.842248_699.1 2024-03-04T18:03:20.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:20.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:20.209 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:20.256 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:20.256 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:20.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:20.327+0000 7fed05c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:20.328+0000 7fed05c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:20.369 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:20Z"} 2024-03-04T18:03:20.369 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:20Z"} 2024-03-04T18:03:20.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:20.370 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:20.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:20.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:03:25.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:25.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:25.371 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:25.418 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:25.418 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:25.478 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:25.478+0000 7fb16f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:25.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:25.479+0000 7fb16f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:25.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:25Z"} 2024-03-04T18:03:25.511 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:25Z"} 2024-03-04T18:03:25.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:25.511 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:25.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:25.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:03:30.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:30.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:30.512 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:30.558 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:30.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:30.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:30.691+0000 7effe0436800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:30.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:30.693+0000 7effe0436800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:30.760 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:30Z"} 2024-03-04T18:03:30.760 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:30Z"} 2024-03-04T18:03:30.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:30.760 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:30.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:30.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:03:35.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:35.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:35.763 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:35.836 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:35.837 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:35.906+0000 7f0bdee12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:35.907+0000 7f0bdee12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:35.930 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:35Z"} 2024-03-04T18:03:35.930 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:35Z"} 2024-03-04T18:03:35.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:35.930 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:35.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:35.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:03:40.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:40.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:40.932 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:41.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:41.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:41.136 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:41.136+0000 7f1ab1012800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:41.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:41.137+0000 7f1ab1012800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:41.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:41Z"} 2024-03-04T18:03:41.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:41Z"} 2024-03-04T18:03:41.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:41.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:41.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:41.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:03:46.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:46.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:46.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:46.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:46.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:46.335 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:46.335+0000 7f00d580f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:46.336 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:46.336+0000 7f00d580f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:46.362 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:46Z"} 2024-03-04T18:03:46.362 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:46Z"} 2024-03-04T18:03:46.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:46.362 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:46.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:46.362 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:03:51.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:51.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:51.364 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:51.433 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:51.433 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:51.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:51.590+0000 7f5e7a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:51.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:51.591+0000 7f5e7a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:51.615 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:51Z"} 2024-03-04T18:03:51.615 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:51Z"} 2024-03-04T18:03:51.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:51.616 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:51.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:51.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:03:56.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:56.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:03:56.617 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:03:56.665 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:03:56.665 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:03:56.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:56.728+0000 7f20b4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:56.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:03:56.729+0000 7f20b4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:03:56.754 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:56Z"} 2024-03-04T18:03:56.755 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:56Z"} 2024-03-04T18:03:56.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:03:56.755 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:56.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:03:56.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:04:01.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:01.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:01.756 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:01.801 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:01.802 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:01.854 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:01.854+0000 7f821f32c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:01.854 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:01.854+0000 7f821f32c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:01.879 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:01Z"} 2024-03-04T18:04:01.879 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:01Z"} 2024-03-04T18:04:01.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:01.879 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:01.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:01.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:04:06.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:06.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:06.881 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:06.928 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:06.928 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:06.990 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:06.990+0000 7f78d4444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:06.991 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:06.991+0000 7f78d4444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:07.018 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:06Z"} 2024-03-04T18:04:07.018 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:06Z"} 2024-03-04T18:04:07.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:07.019 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:07.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:07.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T18:04:12.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:12.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:12.021 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:12.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:12.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:12.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:12.142+0000 7f11c6333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:12.143 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:12.143+0000 7f11c6333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:12.169 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:12Z"} 2024-03-04T18:04:12.169 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:12Z"} 2024-03-04T18:04:12.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:12.170 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:12.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:12.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:04:17.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:17.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:17.171 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:17.218 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:17.218 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:17.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:17.281+0000 7fe345f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:17.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:17.282+0000 7fe345f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:17.305 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:17Z"} 2024-03-04T18:04:17.305 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:17Z"} 2024-03-04T18:04:17.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:17.305 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:17.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:17.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:04:22.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:22.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:22.306 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:22.355 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:22.355 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:22.406 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:22.405+0000 7fce05085800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:22.406 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:22.406+0000 7fce05085800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:22.430 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:22Z"} 2024-03-04T18:04:22.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:22Z"} 2024-03-04T18:04:22.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:22.430 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:22.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:22.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:04:27.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:27.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:27.433 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:27.483 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:27.483 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:27.546 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:27.546+0000 7f7f88f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:27.547 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:27.547+0000 7f7f88f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:27.571 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:27Z"} 2024-03-04T18:04:27.572 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:27Z"} 2024-03-04T18:04:27.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:27.572 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:27.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:27.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:04:32.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:32.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:32.573 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:32.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:32.621 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:32.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:32.682+0000 7fd951d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:32.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:32.683+0000 7fd951d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:32.713 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:32Z"} 2024-03-04T18:04:32.713 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:32Z"} 2024-03-04T18:04:32.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:32.714 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:32.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:32.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T18:04:37.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:37.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:37.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:37.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:37.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:37.821 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:37.821+0000 7fb63791e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:37.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:37.822+0000 7fb63791e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:37.847 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:37Z"} 2024-03-04T18:04:37.847 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:37Z"} 2024-03-04T18:04:37.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:37.848 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:37.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:37.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T18:04:42.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:42.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:42.849 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:42.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:42.902 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:42.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:42.970+0000 7fc4e8a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:42.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:42.971+0000 7fc4e8a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:42.995 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:42Z"} 2024-03-04T18:04:42.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:42Z"} 2024-03-04T18:04:42.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:42.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:42.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:42.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:04:47.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:47.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:47.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:48.049 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:48.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:48.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:48.110+0000 7f0eeb015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:48.111 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:48.111+0000 7f0eeb015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:48.134 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T18:04:48.134 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T18:04:48.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:48.135 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:48.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:48.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:04:53.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:53.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:53.136 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:53.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:53.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:53.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:53.254+0000 7f501ac14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:53.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:53.254+0000 7f501ac14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:53.278 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:53Z"} 2024-03-04T18:04:53.278 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:53Z"} 2024-03-04T18:04:53.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:53.278 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:53.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:53.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:04:58.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:58.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:04:58.280 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:04:58.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:04:58.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:04:58.388 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:58.389+0000 7f7239f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:58.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:04:58.390+0000 7f7239f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:04:58.499 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:58Z"} 2024-03-04T18:04:58.500 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:58Z"} 2024-03-04T18:04:58.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:04:58.500 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:58.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:04:58.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T18:05:03.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:03.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:03.502 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:03.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:03.605+0000 7f056508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:03.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:03.606+0000 7f056508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:03.633 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:03Z"} 2024-03-04T18:05:03.633 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:03Z"} 2024-03-04T18:05:03.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:03.634 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:03.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:03.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:05:08.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:08.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:08.636 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:08.689 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:08.689 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:08.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:08.747+0000 7f9cddc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:08.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:08.748+0000 7f9cddc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:08.774 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:08Z"} 2024-03-04T18:05:08.774 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:08Z"} 2024-03-04T18:05:08.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:08.774 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:08.774 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:08.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:05:13.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:13.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:13.776 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:13.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:13.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:13.884 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:13.883+0000 7fdee4faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:13.884 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:13.884+0000 7fdee4faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:13.916 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:13Z"} 2024-03-04T18:05:13.916 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:13Z"} 2024-03-04T18:05:13.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:13.917 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:13.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:13.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T18:05:18.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:18.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:18.919 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:18.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:18.965 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:19.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:19.031+0000 7f5a454c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:19.032 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:19.032+0000 7f5a454c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:19.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T18:05:19.056 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T18:05:19.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:19.057 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:19.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:19.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:05:24.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:24.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:24.058 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:24.104 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:24.104 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:24.166+0000 7ff88833b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:24.167 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:24.167+0000 7ff88833b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:24.191 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T18:05:24.191 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T18:05:24.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:24.191 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:24.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:24.192 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:05:29.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:29.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:29.193 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:29.241 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:29.241 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:29.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:29.299+0000 7f443dd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:29.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:29.300+0000 7f443dd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:29.322 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T18:05:29.322 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T18:05:29.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:29.323 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:29.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:29.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:05:34.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:34.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:34.324 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:34.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:34.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:34.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:34.433+0000 7fd7fc02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:34.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:34.434+0000 7fd7fc02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:34.456 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:34Z"} 2024-03-04T18:05:34.457 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:34Z"} 2024-03-04T18:05:34.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:34.457 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:34.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:34.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:05:39.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:39.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:39.458 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:39.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:39.506 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:39.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:39.565+0000 7f98a6542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:39.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:39.566+0000 7f98a6542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:39.595 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:39Z"} 2024-03-04T18:05:39.595 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:39Z"} 2024-03-04T18:05:39.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:39.595 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:39.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:39.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:05:44.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:44.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:44.597 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:44.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:44.646 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:44.696 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:44.696+0000 7fbb54f1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:44.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:44.697+0000 7fbb54f1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:44.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:44Z"} 2024-03-04T18:05:44.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:44Z"} 2024-03-04T18:05:44.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.179124_720.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.179124Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:44.720 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:44.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.179124_720.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:44.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.179124_720.1 2024-03-04T18:05:49.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:49.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:49.722 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:49.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:49.770 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:49.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:49.842+0000 7fe474e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:49.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:49.843+0000 7fe474e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:49.875 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:49Z"} 2024-03-04T18:05:49.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:49Z"} 2024-03-04T18:05:49.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:49.875 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:49.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:49.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:05:54.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:54.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:05:54.877 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:05:54.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:05:54.930 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:05:55.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:55.002+0000 7f1473a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:55.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:05:55.003+0000 7f1473a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:05:55.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T18:05:55.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T18:05:55.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:05:55.034 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:55.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:05:55.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:06:00.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:00.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:00.036 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:00.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:00.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:00.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:00.151+0000 7f284ef4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:00.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:00.152+0000 7f284ef4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:00.179 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T18:06:00.179 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T18:06:00.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:00.180 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:00.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:00.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:06:05.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:05.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:05.182 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:05.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:05.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:05.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:05.291+0000 7f5e34811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:05.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:05.291+0000 7f5e34811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:05.321 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T18:06:05.321 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T18:06:05.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:05.321 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:05.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:05.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:06:10.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:10.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:10.324 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:10.372 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:10.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:10.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:10.432+0000 7fe6e9b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:10.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:10.433+0000 7fe6e9b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:10.456 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T18:06:10.457 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T18:06:10.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:10.457 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:10.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:10.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:06:15.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:15.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:15.459 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:15.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:15.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:15.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:15.558+0000 7f0dbc111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:15.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:15.559+0000 7f0dbc111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:15.586 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:15Z"} 2024-03-04T18:06:15.586 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:15Z"} 2024-03-04T18:06:15.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:15.587 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:15.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:15.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:06:20.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:20.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:20.588 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:20.637 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:20.637 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:20.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:20.702+0000 7f6eebb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:20.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:20.703+0000 7f6eebb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:20.733 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:20Z"} 2024-03-04T18:06:20.733 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:20Z"} 2024-03-04T18:06:20.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:20.734 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:20.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:20.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:06:25.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:25.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:25.735 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:25.787 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:25.787 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:25.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:25.844+0000 7f745c211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:25.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:25.845+0000 7f745c211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:25.872 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:25Z"} 2024-03-04T18:06:25.872 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:25Z"} 2024-03-04T18:06:25.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:25.873 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:25.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:25.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:06:30.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:30.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:30.874 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:30.923 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:30.923 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:30.973+0000 7f01c9b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:30.974+0000 7f01c9b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:30.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:30Z"} 2024-03-04T18:06:30.997 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:30Z"} 2024-03-04T18:06:30.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:30.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:30.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:30.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.885240_641.1 2024-03-04T18:06:35.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:35.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:36.000 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:36.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:36.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:36.117+0000 7f4e82333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:36.118+0000 7f4e82333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:36.142 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T18:06:36.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T18:06:36.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:36.143 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:36.143 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:36.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:06:41.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:41.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:41.145 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:41.198 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:41.198 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:41.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:41.270+0000 7f1e70f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:41.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:41.271+0000 7f1e70f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:41.295 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T18:06:41.295 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T18:06:41.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:41.295 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:41.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:41.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T18:06:46.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:46.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:46.297 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:46.348 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:46.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:46.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:46.418+0000 7fbc9772a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:46.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:46.419+0000 7fbc9772a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:46.443 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T18:06:46.443 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T18:06:46.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:46.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:46.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:46.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:06:51.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:51.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:51.445 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:51.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:51.550+0000 7fa682d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:51.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:51.550+0000 7fa682d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:51.576 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:51Z"} 2024-03-04T18:06:51.577 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:51Z"} 2024-03-04T18:06:51.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:51.577 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:51.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:51.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:06:56.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:56.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:06:56.578 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:06:56.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:06:56.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:06:56.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:56.702+0000 7ff86ff2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:56.704 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:06:56.702+0000 7ff86ff2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:06:56.725 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:56Z"} 2024-03-04T18:06:56.726 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:56Z"} 2024-03-04T18:06:56.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:06:56.726 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:56.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:06:56.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:07:01.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:01.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:01.728 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:01.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:01.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:01.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:01.972+0000 7f5a75ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:01.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:01.973+0000 7f5a75ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:02.015 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:01Z"} 2024-03-04T18:07:02.015 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:01Z"} 2024-03-04T18:07:02.015 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:02.015 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:02.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:02.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:07:07.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:07.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:07.018 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:07.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:07.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:07.162 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:07.162+0000 7f6538213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:07.163 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:07.163+0000 7f6538213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:07.188 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T18:07:07.188 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T18:07:07.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:07.189 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:07.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:07.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:07:12.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:12.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:12.191 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:12.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:12.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:12.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:12.349+0000 7f568a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:12.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:12.350+0000 7f568a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:12.380 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T18:07:12.381 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T18:07:12.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:12.381 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:12.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:12.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:07:17.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:17.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:17.383 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:17.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:17.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:17.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:17.489+0000 7f2293733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:17.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:17.490+0000 7f2293733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:17.515 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T18:07:17.515 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T18:07:17.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:17.515 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:17.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:17.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.890340_643.1 2024-03-04T18:07:22.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:22.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:22.517 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:22.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:22.565 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:22.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:22.624+0000 7f02da3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:22.625 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:22.625+0000 7f02da3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T18:07:22.654 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T18:07:22.654 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:22.654 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:22.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:22.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:07:27.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:27.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:27.656 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:27.705 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:27.705 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:27.765 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:27.765+0000 7f134888c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:27.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:27.766+0000 7f134888c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:27.788 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T18:07:27.789 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T18:07:27.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:27.789 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:27.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:27.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:07:32.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:32.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:32.791 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:32.843 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:32.844 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:32.904 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:32.903+0000 7fc61c022800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:32.904 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:32.904+0000 7fc61c022800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:32.938 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:32Z"} 2024-03-04T18:07:32.938 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:32Z"} 2024-03-04T18:07:32.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:32.938 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:32.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:32.938 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:07:37.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:37.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:37.940 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:37.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:37.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:38.052+0000 7f305c542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:38.053+0000 7f305c542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:38.077 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T18:07:38.077 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T18:07:38.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:38.078 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:38.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:38.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:07:43.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:43.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:43.079 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:43.157 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:43.158 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:43.283+0000 7f0c70311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:43.283+0000 7f0c70311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:43.305 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T18:07:43.305 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T18:07:43.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:43.305 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:43.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:43.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T18:07:48.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:48.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:48.307 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:48.353 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:48.353 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:48.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:48.412+0000 7f5a66813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:48.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:48.413+0000 7f5a66813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:48.437 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T18:07:48.437 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T18:07:48.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:48.437 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:48.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:48.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:07:53.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:53.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:53.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:53.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:53.485 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:53.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:53.540+0000 7f8794a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:53.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:53.541+0000 7f8794a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:53.568 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T18:07:53.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T18:07:53.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:53.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:53.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:53.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:07:58.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:58.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:07:58.570 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:07:58.616 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:07:58.616 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:07:58.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:58.678+0000 7feaaba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:58.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:07:58.679+0000 7feaaba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:07:58.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T18:07:58.707 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T18:07:58.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:07:58.707 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:58.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:07:58.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:08:03.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:03.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:03.708 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:03.755 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:03.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:03.826+0000 7f00f8016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:03.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:03.827+0000 7f00f8016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:03.856 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T18:08:03.856 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T18:08:03.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:08:03.856 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:08:03.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:08:03.856 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:08:08.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:08.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:08.859 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:08.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:08.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:08.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:08.984+0000 7f92f602d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:08.985 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:08.985+0000 7f92f602d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:09.014 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T18:08:09.014 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T18:08:09.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T18:08:09.015 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:08:09.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T18:08:09.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:08:14.018 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2024-03-04T18:08:14.019 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-sbnvju-1 2024-03-04T18:08:14.019 DEBUG:boto:Method: POST 2024-03-04T18:08:14.020 DEBUG:boto:Path: / 2024-03-04T18:08:14.020 DEBUG:boto:Data: 2024-03-04T18:08:14.020 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.020 DEBUG:boto:Host: smithi027.front.sepia.ceph.com 2024-03-04T18:08:14.020 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.020 DEBUG:boto:Params: {'RoleName': 'roles-sbnvju-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T18:08:14.020 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:08:14.021 DEBUG:boto:Token: None 2024-03-04T18:08:14.021 DEBUG:boto:CanonicalRequest: POST / host:smithi027.front.sepia.ceph.com:8000 x-amz-date:20240304T180814Z host;x-amz-date b61a4c900621bc06f9dcd08e1f0ed3acdcea5791619daf5a84b2bf995cf880f8 2024-03-04T18:08:14.021 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T180814Z 20240304/front/smithi027/aws4_request 3f0369b9e92d6f8fb09d4ede5e37f73ed8e12ab327a32cc0577a82d07ef42010 2024-03-04T18:08:14.021 DEBUG:boto:Signature: c79a4f75a824a89e18a49ffd24bdb8f0e860c8cbc6d4ef44b186aa26fbb5e7e0 2024-03-04T18:08:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=c79a4f75a824a89e18a49ffd24bdb8f0e860c8cbc6d4ef44b186aa26fbb5e7e0', 'Host': 'smithi027.front.sepia.ceph.com:8000'} 2024-03-04T18:08:14.035 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016dca28602f1e5d5-0065e60e0e-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.036 DEBUG:boto:b'5c62f73f-097e-4a38-aa4f-b2e1b8140933roles-sbnvju-1/arn:aws:iam:::role/roles-sbnvju-12024-03-04T18:08:14.27Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000016dca28602f1e5d5-0065e60e0e-5355-test-zone1' 2024-03-04T18:08:14.037 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-sbnvju-2 2024-03-04T18:08:14.037 DEBUG:boto:Method: POST 2024-03-04T18:08:14.037 DEBUG:boto:Path: / 2024-03-04T18:08:14.037 DEBUG:boto:Data: 2024-03-04T18:08:14.037 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.037 DEBUG:boto:Host: smithi071.front.sepia.ceph.com 2024-03-04T18:08:14.037 DEBUG:boto:Port: 8000 2024-03-04T18:08:14.037 DEBUG:boto:Params: {'RoleName': 'roles-sbnvju-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T18:08:14.037 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:08:14.038 DEBUG:boto:Token: None 2024-03-04T18:08:14.038 DEBUG:boto:CanonicalRequest: POST / host:smithi071.front.sepia.ceph.com:8000 x-amz-date:20240304T180814Z host;x-amz-date d04fcd9ee728b24a7ff09d1b4a3182db06cf2195a3fbf9b6297973719f2dae1f 2024-03-04T18:08:14.038 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T180814Z 20240304/front/smithi071/aws4_request 7108251c592d35cd8b1a5fbf5f4cbc5a2b6d0dbcdfacb0041b9f7ad92a7d16c7 2024-03-04T18:08:14.038 DEBUG:boto:Signature: d50ba8d169de68f46f039a71b8b83c60c8537e2c76f65f8ba051c263fd5e39af 2024-03-04T18:08:14.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=d50ba8d169de68f46f039a71b8b83c60c8537e2c76f65f8ba051c263fd5e39af', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2024-03-04T18:08:14.059 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4c0c3b343e0e784-0065e60e0e-7072-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.059 DEBUG:boto:b'00798078-f3f0-491b-8ab9-2bf858a374f4roles-sbnvju-2/arn:aws:iam:::role/roles-sbnvju-22024-03-04T18:08:14.54Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000e4c0c3b343e0e784-0065e60e0e-7072-test-zone2' 2024-03-04T18:08:14.060 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-sbnvju-3 2024-03-04T18:08:14.060 DEBUG:boto:Method: POST 2024-03-04T18:08:14.060 DEBUG:boto:Path: / 2024-03-04T18:08:14.060 DEBUG:boto:Data: 2024-03-04T18:08:14.060 DEBUG:boto:Headers: {} 2024-03-04T18:08:14.060 DEBUG:boto:Host: smithi027.front.sepia.ceph.com 2024-03-04T18:08:14.060 DEBUG:boto:Port: 8001 2024-03-04T18:08:14.060 DEBUG:boto:Params: {'RoleName': 'roles-sbnvju-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T18:08:14.060 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T18:08:14.060 DEBUG:boto:Token: None 2024-03-04T18:08:14.061 DEBUG:boto:CanonicalRequest: POST / host:smithi027.front.sepia.ceph.com:8001 x-amz-date:20240304T180814Z host;x-amz-date f1a083a55c6b13fc99dc20f05c3d7005f286e4ed7df33639cc55c83b010215ce 2024-03-04T18:08:14.061 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T180814Z 20240304/front/smithi027/aws4_request eed8b8971c6ef7df1fd8a835b8a3fe367c1b412bf987cd9f6716baac0b438580 2024-03-04T18:08:14.061 DEBUG:boto:Signature: 58d00efb17f6a35b0a844711c19974823f93ced6080d84a171eb2b20cb6843c7 2024-03-04T18:08:14.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=58d00efb17f6a35b0a844711c19974823f93ced6080d84a171eb2b20cb6843c7', 'Host': 'smithi027.front.sepia.ceph.com:8001'} 2024-03-04T18:08:14.081 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be3e325851c0a769-0065e60e0e-5358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:08:14.081 DEBUG:boto:b'0666cdb2-02c0-480a-9496-df417fd6630aroles-sbnvju-3/arn:aws:iam:::role/roles-sbnvju-32024-03-04T18:08:14.76Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000be3e325851c0a769-0065e60e0e-5358-test-zone3' 2024-03-04T18:08:14.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.082 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:14.185 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:08:14.205 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:08:14.206 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:14.206 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:08:14.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:14.206 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:14.252 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:14.252 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:14.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:14.304+0000 7f0cf2288800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:14.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:14.305+0000 7f0cf2288800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:14.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T18:08:14.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T18:08:14.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:14.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:14.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:14.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:08:19.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:19.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:19.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:19.387 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:19.387 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:19.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:19.461+0000 7f6d01a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:19.462+0000 7f6d01a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:19.493 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T18:08:19.493 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T18:08:19.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.179124_720.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.179124Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:19.493 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:19.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.179124_720.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:19.493 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.179124_720.1 2024-03-04T18:08:24.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:24.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:24.494 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:24.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:24.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:24.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:24.612+0000 7f5c7594b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:24.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:24.613+0000 7f5c7594b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:24.640 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T18:08:24.640 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T18:08:24.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:24.640 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:24.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:24.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:08:29.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:29.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:29.642 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:29.697 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:29.697 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:29.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:29.767+0000 7fd6c1e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:29.768 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:29.768+0000 7fd6c1e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:29Z"} 2024-03-04T18:08:29.793 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:29Z"} 2024-03-04T18:08:29.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:29.794 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:29.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:29.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:08:34.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:34.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:34.795 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:34.875 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:34.875 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:34.992 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:34.992+0000 7f5875d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:34.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:34.993+0000 7f5875d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:35.020 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:34Z"} 2024-03-04T18:08:35.020 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:34Z"} 2024-03-04T18:08:35.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:35.020 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:35.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:35.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:08:40.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:40.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:40.022 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:40.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:40.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:40.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:40.188+0000 7f5fe4b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:40.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:40.189+0000 7f5fe4b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:40.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T18:08:40.221 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T18:08:40.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:40.222 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:40.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:40.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:08:45.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:45.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:45.224 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:45.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:45.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:45.337 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:45.337+0000 7f79c9926800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:45.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:45.337+0000 7f79c9926800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:45.361 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T18:08:45.362 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T18:08:45.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:45.362 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:45.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:45.362 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T18:08:50.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:50.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:50.364 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:50.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:50.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:50.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:50.477+0000 7f40e6011800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:50.478 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:50.478+0000 7f40e6011800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:50.514 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T18:08:50.514 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T18:08:50.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:50.514 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:50.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:50.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T18:08:55.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:55.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:08:55.516 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:08:55.563 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:08:55.563 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:08:55.636 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:55.636+0000 7fc3edc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:55.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:08:55.637+0000 7fc3edc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:08:55.671 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T18:08:55.671 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T18:08:55.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:08:55.671 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:55.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:08:55.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:09:00.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:00.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:00.673 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:00.719 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:00.719 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:00.774 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:00.774+0000 7f02a4530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:00.775 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:00.775+0000 7f02a4530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:00.801 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T18:09:00.802 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T18:09:00.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:00.802 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:00.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:00.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:09:05.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:05.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:05.804 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:05.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:05.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:05.924 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:05.924+0000 7fba74c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:05.925 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:05.925+0000 7fba74c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:05.960 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T18:09:05.961 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T18:09:05.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:05.961 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:05.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:05.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:09:10.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:10.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:10.962 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:11.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:11.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:11.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:11.058+0000 7f2609a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:11.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:11.059+0000 7f2609a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:11.085 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T18:09:11.085 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T18:09:11.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:11.086 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:11.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:11.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.987908_703.1 2024-03-04T18:09:16.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:16.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:16.088 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:16.141 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:16.141 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:16.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:16.201+0000 7f6371daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:16.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:16.201+0000 7f6371daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T18:09:16.225 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T18:09:16.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:16.225 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:16.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:16.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:09:21.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:21.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:21.226 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:21.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:21.325+0000 7f7150212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:21.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:21.326+0000 7f7150212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:21.355 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T18:09:21.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T18:09:21.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:21.355 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:21.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:21.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:09:26.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:26.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:26.357 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:26.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:26.410 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:26.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:26.472+0000 7f190de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:26.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:26.473+0000 7f190de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:26.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T18:09:26.507 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T18:09:26.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:26.507 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:26.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:26.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:09:31.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:31.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:31.509 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:31.558 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:31.558 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:31.625 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:31.626+0000 7f022c70b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:31.626 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:31.627+0000 7f022c70b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:31.651 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T18:09:31.651 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T18:09:31.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:31.651 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:31.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:31.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:09:36.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:36.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:36.653 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:36.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:36.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:36.775 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:36.775+0000 7f0606824800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:36.776 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:36.776+0000 7f0606824800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:36.807 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T18:09:36.807 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T18:09:36.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:36.807 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:36.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:36.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.838159_698.1 2024-03-04T18:09:41.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:41.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:41.809 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:41.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:41.883 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:42.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:42.035+0000 7fc67d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:42.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:42.036+0000 7fc67d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:42.060 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:42Z"} 2024-03-04T18:09:42.060 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:42Z"} 2024-03-04T18:09:42.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:42.060 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:42.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:42.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:09:47.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:47.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:47.062 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:47.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:47.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:47.166 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:47.166+0000 7ff78d630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:47.167 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:47.167+0000 7ff78d630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:47.194 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T18:09:47.194 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T18:09:47.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:47.195 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:47.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:47.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:09:52.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:52.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:52.196 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:52.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:52.243 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:52.318 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:52.317+0000 7f8828e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:52.318 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:52.318+0000 7f8828e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:52.368 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T18:09:52.368 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T18:09:52.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:52.368 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:52.368 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:52.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:09:57.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:57.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:09:57.370 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:09:57.416 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:09:57.416 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:09:57.484 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:57.484+0000 7f8cc2c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:57.485 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:09:57.485+0000 7f8cc2c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:09:57.514 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T18:09:57.514 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T18:09:57.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:09:57.515 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:57.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:09:57.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.120400_708.1 2024-03-04T18:10:02.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:02.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:02.516 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:02.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:02.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:02.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:02.617+0000 7fa6fb816800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:02.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:02.618+0000 7fa6fb816800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:02.643 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T18:10:02.643 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T18:10:02.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:02.644 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:02.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:02.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:10:07.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:07.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:07.646 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:07.692 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:07.692 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:07.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:07.737+0000 7efdb1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:07.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:07.738+0000 7efdb1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:07.768 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T18:10:07.768 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T18:10:07.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:07.769 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:07.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:07.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:10:12.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:12.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:12.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:12.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:12.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:12.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:12.867+0000 7f23d6731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:12.868 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:12.868+0000 7f23d6731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:12.890 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T18:10:12.890 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T18:10:12.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:12.891 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:12.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:12.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:10:17.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:17.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:17.892 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:17.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:17.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:17.989+0000 7fcd46f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:17.990+0000 7fcd46f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:18.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T18:10:18.016 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T18:10:18.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:18.016 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:18.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:18.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:10:23.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:23.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:23.018 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:23.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:23.064 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:23.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:23.115+0000 7fba3c612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:23.116 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:23.116+0000 7fba3c612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:23.139 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:23Z"} 2024-03-04T18:10:23.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:23Z"} 2024-03-04T18:10:23.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:23.140 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:23.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:23.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T18:10:28.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:28.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:28.143 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:28.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:28.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:28.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:28.244+0000 7f5289a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:28.245+0000 7f5289a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:28.268 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T18:10:28.268 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T18:10:28.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:28.269 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:28.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:28.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T18:10:33.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:33.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:33.270 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:33.320 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:33.320 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:33.386 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:33.386+0000 7fcd27410800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:33.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:33.386+0000 7fcd27410800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:33.410 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T18:10:33.411 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T18:10:33.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:33.411 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:33.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:33.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:10:38.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:38.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:38.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:38.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:38.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:38.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:38.527+0000 7fd624d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:38.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:38.528+0000 7fd624d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:38.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T18:10:38.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T18:10:38.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:38.560 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:38.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:38.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:10:43.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:43.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:43.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:43.616 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:43.616 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:43.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:43.692+0000 7fe29f92c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:43.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:43.693+0000 7fe29f92c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:43.718 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T18:10:43.718 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T18:10:43.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:43.718 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:43.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:43.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:10:48.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:48.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:48.720 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:48.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:48.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:48.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:48.840+0000 7f5001c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:48.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:48.841+0000 7f5001c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:48.865 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T18:10:48.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T18:10:48.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:48.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:48.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:48.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.996460_704.1 2024-03-04T18:10:53.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:53.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:53.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:53.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:53.920 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:53.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:53.998+0000 7fe0c1b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:53.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:53.999+0000 7fe0c1b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:54.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:54Z"} 2024-03-04T18:10:54.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:54Z"} 2024-03-04T18:10:54.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:54.033 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:54.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:54.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:10:59.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:59.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:10:59.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:10:59.087 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:10:59.087 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:10:59.162 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:59.161+0000 7f48052c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:59.163 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:10:59.162+0000 7f48052c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:10:59.189 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:59Z"} 2024-03-04T18:10:59.189 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:59Z"} 2024-03-04T18:10:59.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:10:59.190 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:59.190 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:10:59.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:11:04.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:04.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:04.191 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:04.244 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:04.244 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:04.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:04.297+0000 7f2d7428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:04.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:04.297+0000 7f2d7428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:04.321 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:04Z"} 2024-03-04T18:11:04.321 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:04Z"} 2024-03-04T18:11:04.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:04.321 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:04.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:04.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:11:09.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:09.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:09.323 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:09.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:09.462+0000 7f4cd0d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:09.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:09.465+0000 7f4cd0d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:09.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T18:11:09.568 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T18:11:09.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:09.568 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:09.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:09.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:11:14.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:14.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:14.570 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:14.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:14.696+0000 7f3647533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:14.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:14.697+0000 7f3647533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T18:11:14.724 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T18:11:14.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:14.724 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:14.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:14.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:11:19.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:19.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:19.726 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:19.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:19.772 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:19.829+0000 7fdaf8a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:19.829+0000 7fdaf8a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:19.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T18:11:19.864 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T18:11:19.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:19.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:19.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:19.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:11:24.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:24.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:24.866 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:24.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:24.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:24.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:24.962+0000 7f602be0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:24.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:24.963+0000 7f602be0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:24.989 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T18:11:24.989 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T18:11:24.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:24.990 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:24.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:24.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:11:29.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:29.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:29.991 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:30.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:30.040 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:30.100 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:30.099+0000 7fe590b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:30.100 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:30.100+0000 7fe590b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:30.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:30Z"} 2024-03-04T18:11:30.127 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:30Z"} 2024-03-04T18:11:30.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:30.127 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:30.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:30.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T18:11:35.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:35.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:35.129 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:35.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:35.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:35.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:35.245+0000 7f81c3533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:35.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:35.246+0000 7f81c3533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:35.272 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:35Z"} 2024-03-04T18:11:35.272 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:35Z"} 2024-03-04T18:11:35.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:35.273 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:35.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:35.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:11:40.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:40.274 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:40.325 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:40.325 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:40.395 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:40.394+0000 7fd2c854b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:40.395 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:40.395+0000 7fd2c854b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:40.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:40Z"} 2024-03-04T18:11:40.418 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:40Z"} 2024-03-04T18:11:40.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:40.418 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:40.418 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:40.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:11:45.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:45.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:45.420 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:45.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:45.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:45.518 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:45.517+0000 7f3417a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:45.518 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:45.518+0000 7f3417a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:45.551 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:45Z"} 2024-03-04T18:11:45.551 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:45Z"} 2024-03-04T18:11:45.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:45.552 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:45.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:45.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:11:50.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:50.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:50.553 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:50.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:50.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:50.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:50.659+0000 7f24dcd3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:50.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:50.660+0000 7f24dcd3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:50.689 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T18:11:50.689 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T18:11:50.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:50.689 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:50.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:50.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.116657_707.1 2024-03-04T18:11:55.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:55.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:11:55.692 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:11:55.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:11:55.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:11:55.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:55.800+0000 7fb3caf2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:55.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:11:55.801+0000 7fb3caf2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:11:55.825 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.180479_721.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.180479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T18:11:55.825 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.180479_721.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.180479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T18:11:55.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.180479_721.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.180479Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:11:55.826 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:55.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.180479_721.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:11:55.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.180479_721.1 2024-03-04T18:12:00.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:00.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:00.827 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:00.879 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:00.879 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:00.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:00.946+0000 7f82b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:00.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:00.949+0000 7f82b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:01.028 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T18:12:01.028 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T18:12:01.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:01.028 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:01.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:01.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:12:06.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:06.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:06.030 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:06.112 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:06.113 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:06.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:06.191+0000 7fb9b702e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:06.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:06.192+0000 7fb9b702e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:06.219 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:06Z"} 2024-03-04T18:12:06.220 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:06Z"} 2024-03-04T18:12:06.220 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:06.220 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:06.220 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:06.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:12:11.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:11.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:11.222 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:11.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:11.347+0000 7fa0c352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:11.348 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:11.348+0000 7fa0c352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:11.370 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:11Z"} 2024-03-04T18:12:11.371 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:11Z"} 2024-03-04T18:12:11.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:11.371 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:11.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:11.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T18:12:16.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:16.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:16.373 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:16.419 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:16.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:16.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:16.479+0000 7ff61a615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:16.480 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:16.480+0000 7ff61a615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:16.502 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:16Z"} 2024-03-04T18:12:16.503 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:16Z"} 2024-03-04T18:12:16.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:16.503 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:16.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:16.503 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T18:12:21.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:21.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:21.505 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:21.552 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:21.553 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:21.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:21.621+0000 7f4581624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:21.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:21.622+0000 7f4581624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:21.654 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:21Z"} 2024-03-04T18:12:21.654 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:21Z"} 2024-03-04T18:12:21.654 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:21.654 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:21.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:21.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T18:12:26.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:26.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:26.656 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:26.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:26.703 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:26.757+0000 7fb9ea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:26.757+0000 7fb9ea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:26.779 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T18:12:26.780 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T18:12:26.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:26.780 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:26.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:26.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T18:12:31.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:31.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:31.782 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:31.829 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:31.829 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:31.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:31.885+0000 7f9b69326800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:31.886 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:31.886+0000 7f9b69326800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:31.908 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T18:12:31.908 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T18:12:31.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:31.909 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:31.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:31.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:12:36.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:36.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:36.910 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:36.977 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:36.977 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:37.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:37.061+0000 7f5baf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:37.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:37.062+0000 7f5baf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:37.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:37Z"} 2024-03-04T18:12:37.099 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:37Z"} 2024-03-04T18:12:37.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:37.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:37.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:37.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:12:42.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:42.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:42.101 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:42.147 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:42.147 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:42.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:42.205+0000 7f415240f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:42.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:42.206+0000 7f415240f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:42.232 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:42Z"} 2024-03-04T18:12:42.232 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:42Z"} 2024-03-04T18:12:42.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:42.232 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:42.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:42.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:12:47.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:47.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:47.234 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:47.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:47.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:47.327 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:47.327+0000 7f911842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:47.328 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:47.328+0000 7f911842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:47.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:47Z"} 2024-03-04T18:12:47.351 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:47Z"} 2024-03-04T18:12:47.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:47.351 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:47.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:47.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:12:52.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:52.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:52.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:52.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:52.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:52.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:52.461+0000 7fd914a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:52.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:52.462+0000 7fd914a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:52.490 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:52Z"} 2024-03-04T18:12:52.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:52Z"} 2024-03-04T18:12:52.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:52.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:52.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:52.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:12:57.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:57.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:12:57.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:12:57.538 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:12:57.538 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:12:57.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:57.586+0000 7fad52733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:57.587 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:12:57.587+0000 7fad52733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:12:57.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:57Z"} 2024-03-04T18:12:57.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:57Z"} 2024-03-04T18:12:57.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:12:57.610 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:57.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:12:57.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:13:02.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:02.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:02.612 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:02.658 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:02.658 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:02.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:02.712+0000 7f2e1648c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:02.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:02.713+0000 7f2e1648c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:02.848 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:02Z"} 2024-03-04T18:13:02.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:02Z"} 2024-03-04T18:13:02.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:02.849 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:02.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:02.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:13:07.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:07.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:07.851 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:07.897 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:07.897 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:07.955 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:07.955+0000 7febdf88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:07.956 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:07.956+0000 7febdf88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:07.983 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T18:13:07.983 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T18:13:07.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:07.983 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:07.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:07.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T18:13:12.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:12.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:12.985 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:13.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:13.033 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:13.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:13.098+0000 7fbd6a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:13.098 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:13.099+0000 7fbd6a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:13.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:13Z"} 2024-03-04T18:13:13.126 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:13Z"} 2024-03-04T18:13:13.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:13.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:13.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:13.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:13:18.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:18.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:18.127 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:18.179 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:18.179 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:18.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:18.238+0000 7fb437615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:18.239+0000 7fb437615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:18.274 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:18Z"} 2024-03-04T18:13:18.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:18Z"} 2024-03-04T18:13:18.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:18.274 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:18.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:18.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:13:23.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... FAIL 2024-03-04T18:13:23.279 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-sbnvju-4 2024-03-04T18:13:23.279 DEBUG:boto:Method: POST 2024-03-04T18:13:23.279 DEBUG:boto:Path: / 2024-03-04T18:13:23.280 DEBUG:boto:Data: 2024-03-04T18:13:23.280 DEBUG:boto:Headers: {} 2024-03-04T18:13:23.280 DEBUG:boto:Host: smithi027.front.sepia.ceph.com 2024-03-04T18:13:23.280 DEBUG:boto:Port: 8000 2024-03-04T18:13:23.280 DEBUG:boto:Params: {'RoleName': 'roles-sbnvju-4', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T18:13:23.281 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:13:23.281 DEBUG:boto:Token: None 2024-03-04T18:13:23.281 DEBUG:boto:CanonicalRequest: POST / host:smithi027.front.sepia.ceph.com:8000 x-amz-date:20240304T181323Z host;x-amz-date 6288b3b630f5b40a8492676736e698fd56289399f9976b986128f5ed073c2ef1 2024-03-04T18:13:23.281 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240304T181323Z 20240304/front/smithi027/aws4_request 89a7f99f6174137c51f75b45970cf9eddbd72c1033c261d7de8225d10741b841 2024-03-04T18:13:23.282 DEBUG:boto:Signature: 2160f08a166171f21c6c6bed5e28ec21af78c4255f898ccbb70ebe7596b282e6 2024-03-04T18:13:23.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T181323Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=2160f08a166171f21c6c6bed5e28ec21af78c4255f898ccbb70ebe7596b282e6', 'Host': 'smithi027.front.sepia.ceph.com:8000'} 2024-03-04T18:13:23.296 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da1f61449bdbdd2c-0065e60f43-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '731'), ('Date', 'Mon, 04 Mar 2024 18:13:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:13:23.296 DEBUG:boto:b'0a1839a4-a8ef-4bdb-bf5e-1055e6056fb3roles-sbnvju-4/arn:aws:iam:::role/roles-sbnvju-42024-03-04T18:13:23.287Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000da1f61449bdbdd2c-0065e60f43-5355-test-zone1' 2024-03-04T18:13:23.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:23.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:23.298 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:13:23.422 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:13:23.422 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:23.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:13:23.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:23.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:23.423 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:23.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:23.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:23.535 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:23.534+0000 7f6ab1a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:23.537 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:23.537+0000 7f6ab1a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:23.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:23Z"} 2024-03-04T18:13:23.564 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:23Z"} 2024-03-04T18:13:23.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:23.564 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:23.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:23.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:13:28.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:28.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:28.566 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:28.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:28.684+0000 7efdd9f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:28.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:28.685+0000 7efdd9f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:28.713 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:28Z"} 2024-03-04T18:13:28.713 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:28Z"} 2024-03-04T18:13:28.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:28.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:28.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:28.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.242177_729.1 2024-03-04T18:13:33.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:33.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:33.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:33.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:33.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:33.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:33.930+0000 7f6046c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:33.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:33.936+0000 7f6046c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:33Z"} 2024-03-04T18:13:33.999 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:33Z"} 2024-03-04T18:13:33.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:33.999 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:33.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:33.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:13:39.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:39.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:39.001 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:39.061 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:39.061 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:39.113 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:39.113+0000 7f1975a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:39.114 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:39.114+0000 7f1975a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:39.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:39Z"} 2024-03-04T18:13:39.145 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:39Z"} 2024-03-04T18:13:39.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:39.146 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:39.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:39.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:13:44.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:44.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:44.147 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:44.256+0000 7f5e79730800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:44.257+0000 7f5e79730800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:44.279 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:44Z"} 2024-03-04T18:13:44.279 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:44Z"} 2024-03-04T18:13:44.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:44.280 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:44.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:44.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:13:49.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:49.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:49.282 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:49.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:49.327 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:49.389 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:49.389+0000 7f7351f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:49.390 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:49.390+0000 7f7351f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:49.417 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:49Z"} 2024-03-04T18:13:49.418 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:49Z"} 2024-03-04T18:13:49.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:49.418 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:49.418 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:49.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:13:54.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:54.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:54.420 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:54.468 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:54.469 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:54.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:54.516+0000 7fc73f624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:54.517+0000 7fc73f624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:54.539 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:54Z"} 2024-03-04T18:13:54.540 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:54Z"} 2024-03-04T18:13:54.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:54.540 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:54.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:54.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:13:59.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:59.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:13:59.542 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:13:59.588 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:13:59.588 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:13:59.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:59.649+0000 7f3b47ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:59.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:13:59.650+0000 7f3b47ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:13:59.672 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:59Z"} 2024-03-04T18:13:59.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:59Z"} 2024-03-04T18:13:59.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:13:59.673 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:59.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:13:59.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:14:04.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:04.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:04.676 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:04.723 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:04.723 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:04.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:04.785+0000 7f1951844800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:04.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:04.786+0000 7f1951844800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:04.820 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:04Z"} 2024-03-04T18:14:04.821 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:04Z"} 2024-03-04T18:14:04.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:04.821 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:04.821 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:04.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T18:14:09.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:09.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:09.823 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:09.870 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:09.870 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:09.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:09.919+0000 7f32fcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:09.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:09.919+0000 7f32fcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:09.942 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:09Z"} 2024-03-04T18:14:09.942 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:09Z"} 2024-03-04T18:14:09.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:09.943 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:09.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:09.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:14:14.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:14.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:14.946 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:14.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:14.996 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:15.051 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:15.051+0000 7f6b6b013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:15.051 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:15.052+0000 7f6b6b013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:15.074 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:15Z"} 2024-03-04T18:14:15.074 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:15Z"} 2024-03-04T18:14:15.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:15.074 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:15.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:15.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:14:20.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:20.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:20.076 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:20.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:20.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:20.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:20.189+0000 7ff59c68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:20.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:20.190+0000 7ff59c68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:20.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:20Z"} 2024-03-04T18:14:20.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:20Z"} 2024-03-04T18:14:20.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:20.214 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:20.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:20.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:14:25.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:25.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:25.217 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:25.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:25.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:25.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:25.332+0000 7f3328e42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:25.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:25.333+0000 7f3328e42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:25.358 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:25Z"} 2024-03-04T18:14:25.358 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:25Z"} 2024-03-04T18:14:25.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:25.359 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:25.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:25.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:14:30.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:30.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:30.361 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:30.414 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:30.414 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:30.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:30.466+0000 7f127ac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:30.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:30.467+0000 7f127ac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:30.490 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:30Z"} 2024-03-04T18:14:30.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:30Z"} 2024-03-04T18:14:30.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:30.491 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:30.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:30.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:14:35.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:35.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:35.493 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:35.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:35.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:35.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:35.599+0000 7f3269f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:35.600 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:35.600+0000 7f3269f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:35.624 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:35Z"} 2024-03-04T18:14:35.624 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:35Z"} 2024-03-04T18:14:35.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:35.625 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:35.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:35.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:14:40.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:40.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:40.626 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:40.677 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:40.677 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:40.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:40.728+0000 7f11f4b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:40.729+0000 7f11f4b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:40.756 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:40Z"} 2024-03-04T18:14:40.756 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:40Z"} 2024-03-04T18:14:40.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:40.757 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:40.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:40.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T18:14:45.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:45.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:45.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:45.812 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:45.812 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:45.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:45.874+0000 7fa6e2111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:45.875 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:45.875+0000 7fa6e2111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:45.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:45Z"} 2024-03-04T18:14:45.902 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:45Z"} 2024-03-04T18:14:45.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:45.902 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:45.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:45.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T18:14:50.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:50.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:50.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:50.957 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:50.957 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:51.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:51.005+0000 7fd061d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:51.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:51.006+0000 7fd061d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:51.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:51Z"} 2024-03-04T18:14:51.031 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:51Z"} 2024-03-04T18:14:51.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:51.031 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:51.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:51.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:14:56.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:56.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:14:56.033 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:14:56.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:14:56.085 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:14:56.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:56.158+0000 7f5df1212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:56.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:14:56.159+0000 7f5df1212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:14:56.193 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:56Z"} 2024-03-04T18:14:56.193 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:56Z"} 2024-03-04T18:14:56.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:14:56.193 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:56.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:14:56.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:15:01.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:01.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:01.195 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:01.278 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:01.278 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:01.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:01.366+0000 7fd06f622800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:01.367 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:01.367+0000 7fd06f622800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:01.391 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:01Z"} 2024-03-04T18:15:01.391 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:01Z"} 2024-03-04T18:15:01.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:01.392 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:01.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:01.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:15:06.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:06.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:06.394 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:06.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:06.502+0000 7f92bfe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:06.503 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:06.503+0000 7f92bfe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:06.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:06Z"} 2024-03-04T18:15:06.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:06Z"} 2024-03-04T18:15:06.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:06.529 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:06.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:06.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:15:11.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:11.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:11.531 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:11.576 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:11.577 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:11.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:11.634+0000 7f498d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:11.635 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:11.635+0000 7f498d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:11.657 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:11Z"} 2024-03-04T18:15:11.658 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:11Z"} 2024-03-04T18:15:11.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:11.658 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:11.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:11.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:15:16.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:16.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:16.660 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:16.707 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:16.707 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:16.765+0000 7f9f4e244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:16.766+0000 7f9f4e244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:16.792 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:16Z"} 2024-03-04T18:15:16.793 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:16Z"} 2024-03-04T18:15:16.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:16.793 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:16.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:16.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:15:21.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:21.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:21.795 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:21.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:21.843 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:21.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:21.894+0000 7f353b12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:21.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:21.895+0000 7f353b12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:21.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:21Z"} 2024-03-04T18:15:21.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:21Z"} 2024-03-04T18:15:21.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:21.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:21.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:21.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T18:15:26.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:26.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:26.926 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:26.973 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:26.973 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:27.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:27.025+0000 7fd04e211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:27.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:27.026+0000 7fd04e211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:27.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.343580_740.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.343580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:27Z"} 2024-03-04T18:15:27.049 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.343580_740.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.343580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:27Z"} 2024-03-04T18:15:27.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.343580_740.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.343580Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:27.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:27.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.343580_740.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:27.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571980.343580_740.1 2024-03-04T18:15:32.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:32.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:32.051 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:32.102 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:32.102 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:32.155+0000 7f08a3f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:32.156+0000 7f08a3f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:32.178 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:32Z"} 2024-03-04T18:15:32.178 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:32Z"} 2024-03-04T18:15:32.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:32.179 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:32.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:32.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:15:37.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:37.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:37.181 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:37.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:37.231 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:37.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:37.290+0000 7fc4ebba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:37.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:37.291+0000 7fc4ebba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:37.315 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:37Z"} 2024-03-04T18:15:37.316 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:37Z"} 2024-03-04T18:15:37.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:37.316 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:37.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:37.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:15:42.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:42.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:42.318 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:42.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:42.366 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:42.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:42.435+0000 7fea12013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:42.436 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:42.436+0000 7fea12013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:42.460 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:42Z"} 2024-03-04T18:15:42.461 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:42Z"} 2024-03-04T18:15:42.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:42.461 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:42.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:42.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:15:47.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:47.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:47.463 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:47.512 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:47.512 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:47.582 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:47.582+0000 7ff2a8614800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:47.583 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:47.583+0000 7ff2a8614800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:47.613 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:47Z"} 2024-03-04T18:15:47.613 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:47Z"} 2024-03-04T18:15:47.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:47.613 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:47.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:47.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T18:15:52.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:52.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:52.615 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:52.667 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:52.667 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:52.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:52.721+0000 7f4f954c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:52.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:52.722+0000 7f4f954c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:52.752 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:52Z"} 2024-03-04T18:15:52.752 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:52Z"} 2024-03-04T18:15:52.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:52.753 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:52.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:52.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:15:57.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:57.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:15:57.754 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:15:57.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:15:57.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:15:57.964 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:57.964+0000 7f570192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:57.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:15:57.968+0000 7f570192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:15:58.019 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:57Z"} 2024-03-04T18:15:58.019 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:57Z"} 2024-03-04T18:15:58.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:15:58.019 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:58.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:15:58.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:16:03.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:03.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:03.021 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:03.121+0000 7f6747a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:03.122+0000 7f6747a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:03.144 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:03Z"} 2024-03-04T18:16:03.144 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:03Z"} 2024-03-04T18:16:03.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:03.144 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:03.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:03.145 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T18:16:08.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:08.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:08.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:08.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:08.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:08.256 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:08.256+0000 7f973ef42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:08.256 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:08.257+0000 7f973ef42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:08.290 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:08Z"} 2024-03-04T18:16:08.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:08Z"} 2024-03-04T18:16:08.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:08.290 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:08.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:08.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:16:13.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:13.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:13.292 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:13.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:13.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:13.406 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:13.406+0000 7f01fc922800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:13.407 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:13.407+0000 7f01fc922800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:13.429 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:13Z"} 2024-03-04T18:16:13.430 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:13Z"} 2024-03-04T18:16:13.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:13.431 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:13.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:13.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:16:18.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:18.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:18.432 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:18.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:18.479 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:18.544 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:18.544+0000 7fac14b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:18.545 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:18.545+0000 7fac14b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:18.569 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:18Z"} 2024-03-04T18:16:18.569 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:18Z"} 2024-03-04T18:16:18.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:18.569 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:18.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:18.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T18:16:23.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:23.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:23.571 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:23.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:23.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:23.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:23.676+0000 7f1ec7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:23.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:23.677+0000 7f1ec7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:23.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:23Z"} 2024-03-04T18:16:23.702 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:23Z"} 2024-03-04T18:16:23.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:23.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:23.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:23.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:16:28.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:28.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:28.705 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:28.760 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:28.760 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:28.830+0000 7f6f9b53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:28.831+0000 7f6f9b53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:28.854 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:28Z"} 2024-03-04T18:16:28.854 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:28Z"} 2024-03-04T18:16:28.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:28.854 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:28.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:28.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T18:16:33.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:33.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:33.856 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:33.911 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:33.911 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:33.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:33.961+0000 7fc537f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:33.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:33.962+0000 7fc537f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:33.985 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:33Z"} 2024-03-04T18:16:33.985 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:33Z"} 2024-03-04T18:16:33.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:33.986 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:33.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:33.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:16:38.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:38.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:38.987 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:39.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:39.044 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:39.105 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:39.106+0000 7efe3d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:39.106 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:39.107+0000 7efe3d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:39.137 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:39Z"} 2024-03-04T18:16:39.138 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:39Z"} 2024-03-04T18:16:39.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:39.138 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:39.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:39.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T18:16:44.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:44.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:44.140 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:44.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:44.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:44.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:44.240+0000 7f0ba8b21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:44.241+0000 7f0ba8b21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:44.267 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:44Z"} 2024-03-04T18:16:44.267 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:44Z"} 2024-03-04T18:16:44.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:44.268 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:44.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:44.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:16:49.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:49.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:49.270 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:49.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:49.317 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:49.371 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:49.372+0000 7fa834413800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:49.372 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:49.373+0000 7fa834413800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:49.394 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:49Z"} 2024-03-04T18:16:49.395 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:49Z"} 2024-03-04T18:16:49.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:49.395 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:49.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:49.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T18:16:54.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:54.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:54.397 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:54.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:54.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:54.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:54.498+0000 7ffa38430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:54.497 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:54.499+0000 7ffa38430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:54.527 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:54Z"} 2024-03-04T18:16:54.527 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:54Z"} 2024-03-04T18:16:54.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:54.527 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:54.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:54.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:16:59.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:59.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:16:59.529 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:16:59.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:16:59.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:16:59.628 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:59.628+0000 7f4ae309d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:59.629 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:16:59.629+0000 7f4ae309d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:16:59.651 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:59Z"} 2024-03-04T18:16:59.652 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:59Z"} 2024-03-04T18:16:59.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:16:59.652 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:59.652 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:16:59.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T18:17:04.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:04.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:04.654 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:04.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:04.700 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:04.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:04.752+0000 7fda6b311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:04.752 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:04.753+0000 7fda6b311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:04.774 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:04Z"} 2024-03-04T18:17:04.774 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:04Z"} 2024-03-04T18:17:04.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:04.775 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:04.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:04.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T18:17:09.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:09.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:09.776 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:09.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:09.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:09.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:09.879+0000 7f693a242800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:09.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:09.880+0000 7f693a242800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:09.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:09Z"} 2024-03-04T18:17:09.902 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:09Z"} 2024-03-04T18:17:09.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:09.902 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:09.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:09.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:17:14.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:14.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:14.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:14.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:14.958 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:15.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:15.036+0000 7f935ea1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:15.037 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:15.037+0000 7f935ea1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:15.063 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:15Z"} 2024-03-04T18:17:15.063 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:15Z"} 2024-03-04T18:17:15.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:15.064 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:15.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:15.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:17:20.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:20.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:20.065 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:20.113 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:20.113 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:20.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:20.176+0000 7fcba8414800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:20.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:20.177+0000 7fcba8414800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:20.201 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:20Z"} 2024-03-04T18:17:20.201 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:20Z"} 2024-03-04T18:17:20.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:20.202 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:20.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:20.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:17:25.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:25.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:25.203 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:25.252 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:25.252 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:25.317 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:25.317+0000 7f188ea8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:25.318 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:25.318+0000 7f188ea8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:25.343 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:25Z"} 2024-03-04T18:17:25.343 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:25Z"} 2024-03-04T18:17:25.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:25.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:25.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:25.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T18:17:30.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:30.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:30.345 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:30.458 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:30.458+0000 7fed62613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:30.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:30.460+0000 7fed62613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:30.483 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:30Z"} 2024-03-04T18:17:30.483 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:30Z"} 2024-03-04T18:17:30.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:30.484 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:30.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:30.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:17:35.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:35.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:35.486 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:35.536 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:35.536 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:35.630+0000 7fb82a68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:35.631 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:35.631+0000 7fb82a68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:35.740 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:35Z"} 2024-03-04T18:17:35.741 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:35Z"} 2024-03-04T18:17:35.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:35.741 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:35.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:35.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T18:17:40.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:40.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:40.742 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:40.839 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:40.839+0000 7f2736015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:40.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:40.840+0000 7f2736015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:40.870 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:40Z"} 2024-03-04T18:17:40.871 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:40Z"} 2024-03-04T18:17:40.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:40.871 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:40.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:40.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:17:45.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:45.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:45.873 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:45.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:45.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:45.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:45.968+0000 7f401500f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:45.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:45.969+0000 7f401500f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:45.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:45Z"} 2024-03-04T18:17:45.994 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:45Z"} 2024-03-04T18:17:45.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:45.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:45.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:45.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T18:17:50.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:50.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:50.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:51.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:51.094+0000 7f9cc5c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:51.094 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:51.095+0000 7f9cc5c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:51.117 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:51Z"} 2024-03-04T18:17:51.118 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:51Z"} 2024-03-04T18:17:51.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:51.118 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:51.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:51.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T18:17:56.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:56.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:17:56.121 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:17:56.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:17:56.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:17:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:56.237+0000 7f4f0cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:17:56.238+0000 7f4f0cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:17:56.270 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:56Z"} 2024-03-04T18:17:56.271 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:56Z"} 2024-03-04T18:17:56.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:17:56.271 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:56.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:17:56.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.002120_706.1 2024-03-04T18:18:01.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:01.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:01.273 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:01.319 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:01.319 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:01.390 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:01.390+0000 7f5baec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:01.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:01.391+0000 7f5baec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:01.413 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:01Z"} 2024-03-04T18:18:01.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:01Z"} 2024-03-04T18:18:01.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:01.414 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:01.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:01.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T18:18:06.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:06.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:06.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:06.463 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:06.464 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:06.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:06.528+0000 7f71ce733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:06.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:06.529+0000 7f71ce733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:06.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:06Z"} 2024-03-04T18:18:06.561 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:06Z"} 2024-03-04T18:18:06.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:06.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:06.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:06.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T18:18:11.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:11.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:11.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:11.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:11.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:11.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:11.662+0000 7fd54c2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:11.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:11.663+0000 7fd54c2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:11.689 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:11Z"} 2024-03-04T18:18:11.689 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:11Z"} 2024-03-04T18:18:11.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:11.689 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:11.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:11.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T18:18:16.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:16.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:16.691 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:16.738 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:16.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:16.810+0000 7fdd3a22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:16.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:16.811+0000 7fdd3a22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:16.838 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:16Z"} 2024-03-04T18:18:16.838 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:16Z"} 2024-03-04T18:18:16.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:16.839 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:16.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:16.839 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T18:18:21.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:21.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:21.840 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:21.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:21.888 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:21.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:21.951+0000 7f131b92d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:21.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:21.952+0000 7f131b92d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:21.980 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:21Z"} 2024-03-04T18:18:21.980 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:21Z"} 2024-03-04T18:18:21.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:21.981 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:21.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:21.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T18:18:26.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:26.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:26.982 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:27.028 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:27.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:27.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:27.077+0000 7f0986739800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:27.078 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:27.078+0000 7f0986739800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:27.101 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:27Z"} 2024-03-04T18:18:27.101 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:27Z"} 2024-03-04T18:18:27.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:27.102 DEBUG:rgw_multi.tests:log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:27.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:27.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T18:18:32.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_delete_sync ... FAIL 2024-03-04T18:18:32.106 DEBUG:boto:path=/sbnvju-74/ 2024-03-04T18:18:32.106 DEBUG:boto:auth_path=/sbnvju-74/ 2024-03-04T18:18:32.106 DEBUG:boto:Method: PUT 2024-03-04T18:18:32.106 DEBUG:boto:Path: /sbnvju-74/ 2024-03-04T18:18:32.107 DEBUG:boto:Data: 2024-03-04T18:18:32.107 DEBUG:boto:Headers: {} 2024-03-04T18:18:32.107 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:18:32.107 DEBUG:boto:Port: 8000 2024-03-04T18:18:32.107 DEBUG:boto:Params: {} 2024-03-04T18:18:32.107 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:18:32.108 DEBUG:boto:Token: None 2024-03-04T18:18:32.108 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:18:32 GMT /sbnvju-74/ 2024-03-04T18:18:32.108 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:VzYo2UVrSAkf8Qn2Tq3Ic+Nxyts= 2024-03-04T18:18:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:18:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VzYo2UVrSAkf8Qn2Tq3Ic+Nxyts=', 'Content-Length': '0'} 2024-03-04T18:18:32.131 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db047ca21b169f9e-0065e61078-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:18:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:18:32.132 DEBUG:rgw_multi.tests:created bucket=sbnvju-74 2024-03-04T18:18:32.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:32.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:32.132 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:18:32.260 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709576312.130624_1055.1","last_update":"2024-03-04T18:18:32.130624Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:18:32.260 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:32.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:18:32.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:32.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:32.261 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:32.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:32.333 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:32.434 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:32.435+0000 7fbe64d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:32.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:32.436+0000 7fbe64d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:32.458 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:32Z"} 2024-03-04T18:18:32.459 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:32Z"} 2024-03-04T18:18:32.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:32.459 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:32.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:32.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T18:18:37.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:37.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:37.461 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:37.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:37.528 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:37.584 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:37.584+0000 7f1cdc28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:37.585 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:37.585+0000 7f1cdc28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:37.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:37Z"} 2024-03-04T18:18:37.609 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:37Z"} 2024-03-04T18:18:37.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:37.610 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:37.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:37.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T18:18:42.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:42.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:42.611 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:42.674 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:42.674 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:42.751 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:42.751+0000 7f1276335800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:42.752 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:42.752+0000 7f1276335800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:42.783 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:42Z"} 2024-03-04T18:18:42.783 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:42Z"} 2024-03-04T18:18:42.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:42.783 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:42.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:42.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T18:18:47.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:47.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:47.784 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:47.839 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:47.839 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:47.893 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:47.894+0000 7ff8dc830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:47.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:47.895+0000 7ff8dc830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:47.918 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:47Z"} 2024-03-04T18:18:47.919 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:47Z"} 2024-03-04T18:18:47.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:47.919 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:47.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:47.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T18:18:52.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:52.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:52.921 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:52.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:52.969 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:53.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:53.019+0000 7f508e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:53.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:53.020+0000 7f508e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:53.042 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:53Z"} 2024-03-04T18:18:53.042 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:53Z"} 2024-03-04T18:18:53.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:53.042 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:53.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:53.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:18:58.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:58.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:18:58.045 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:18:58.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:18:58.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:18:58.167 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:58.168+0000 7f82171a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:58.168 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:18:58.169+0000 7f82171a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:18:58.196 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:58Z"} 2024-03-04T18:18:58.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:58Z"} 2024-03-04T18:18:58.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:18:58.197 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:58.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:18:58.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T18:19:03.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:03.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:03.200 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:03.342+0000 7f57bc08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:03.344+0000 7f57bc08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:03.431 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:03Z"} 2024-03-04T18:19:03.432 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:03Z"} 2024-03-04T18:19:03.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:03.432 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:03.432 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:03.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.002120_706.1 2024-03-04T18:19:08.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:08.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:08.433 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:08.479 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:08.480 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:08.533 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:08.533+0000 7f7fc3424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:08.534 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:08.534+0000 7f7fc3424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:08.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:08Z"} 2024-03-04T18:19:08.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:08Z"} 2024-03-04T18:19:08.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:08.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:08.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:08.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T18:19:13.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:13.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:13.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:13.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:13.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:13.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:13.677+0000 7f00e2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:13.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:13.678+0000 7f00e2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:13.700 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:13Z"} 2024-03-04T18:19:13.700 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:13Z"} 2024-03-04T18:19:13.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:13.700 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:13.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:13.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.692350_570.1 2024-03-04T18:19:18.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:18.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:18.702 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:18.753 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:18.753 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:18.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:18.809+0000 7f03f9d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:18.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:18.809+0000 7f03f9d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:18.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:18Z"} 2024-03-04T18:19:18.833 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:18Z"} 2024-03-04T18:19:18.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:18.834 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:18.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:18.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:19:23.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:23.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:23.835 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:23.884 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:23.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:23.942+0000 7f797bda2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:23.943+0000 7f797bda2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:23.965 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:23Z"} 2024-03-04T18:19:23.965 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:23Z"} 2024-03-04T18:19:23.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:23.965 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:23.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:23.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T18:19:28.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:28.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:28.967 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:29.018 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:29.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:29.072+0000 7fc3da92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:29.073+0000 7fc3da92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:29.095 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:29Z"} 2024-03-04T18:19:29.096 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:29Z"} 2024-03-04T18:19:29.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:29.096 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:29.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:29.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T18:19:34.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:34.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:34.098 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:34.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:34.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:34.220 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:34.221+0000 7f5897267800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:34.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:34.222+0000 7f5897267800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:34.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:34Z"} 2024-03-04T18:19:34.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:34Z"} 2024-03-04T18:19:34.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:34.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:34.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:34.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.608330_559.1 2024-03-04T18:19:39.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:39.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:39.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:39.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:39.359+0000 7f3702931800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:39.358 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:39.360+0000 7f3702931800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:39.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:39Z"} 2024-03-04T18:19:39.386 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:39Z"} 2024-03-04T18:19:39.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:39.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:39.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:39.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.689443_569.1 2024-03-04T18:19:44.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:44.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:44.389 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:44.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:44.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:44.494 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:44.495+0000 7f568072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:44.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:44.496+0000 7f568072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:44.520 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:44Z"} 2024-03-04T18:19:44.520 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:44Z"} 2024-03-04T18:19:44.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:44.521 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:44.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:44.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.684679_567.1 2024-03-04T18:19:49.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:49.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:49.522 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:49.572 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:49.572 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:49.631 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:49.631+0000 7f2699813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:49.632 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:49.632+0000 7f2699813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:49.659 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:49Z"} 2024-03-04T18:19:49.659 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:49Z"} 2024-03-04T18:19:49.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:49.660 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:49.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:49.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:19:54.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:54.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:54.662 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:54.708 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:54.708 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:54.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:54.760+0000 7f60cc0c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:54.760 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:54.761+0000 7f60cc0c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:54Z"} 2024-03-04T18:19:54.784 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:54Z"} 2024-03-04T18:19:54.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:19:54.785 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:54.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:19:54.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:19:59.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:59.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:19:59.787 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:19:59.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:19:59.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:19:59.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:59.997+0000 7f0c5bc0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:19:59.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:19:59.999+0000 7f0c5bc0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:00.052 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:00Z"} 2024-03-04T18:20:00.052 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:00Z"} 2024-03-04T18:20:00.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:00.052 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:00.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:00.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:20:05.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:05.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:05.054 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:05.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:05.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:05.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:05.260+0000 7f6fcf722800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:05.260 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:05.261+0000 7f6fcf722800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:05.286 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:05Z"} 2024-03-04T18:20:05.286 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:05Z"} 2024-03-04T18:20:05.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:05.287 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:05.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:05.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:20:10.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:10.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:10.288 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:10.334 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:10.335 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:10.398 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:10.398+0000 7f2c2732f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:10.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:10.399+0000 7f2c2732f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:10.423 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:10Z"} 2024-03-04T18:20:10.423 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:10Z"} 2024-03-04T18:20:10.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:10.424 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:10.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:10.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.125073_710.1 2024-03-04T18:20:15.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:15.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:15.425 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:15.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:15.536+0000 7f339b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:15.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:15.537+0000 7f339b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:15.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:15Z"} 2024-03-04T18:20:15.564 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:15Z"} 2024-03-04T18:20:15.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:15.564 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:15.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:15.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:20:20.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:20.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:20.566 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:20.612 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:20.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:20.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:20.674+0000 7f695af30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:20.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:20.675+0000 7f695af30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:20.703 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:20Z"} 2024-03-04T18:20:20.703 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:20Z"} 2024-03-04T18:20:20.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:20.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:20.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:20.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.689443_569.1 2024-03-04T18:20:25.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:25.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:25.705 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:25.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:25.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:25.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:25.815+0000 7f12fda0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:25.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:25.816+0000 7f12fda0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:25.839 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:25Z"} 2024-03-04T18:20:25.839 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:25Z"} 2024-03-04T18:20:25.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:25.840 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:25.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:25.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T18:20:30.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:30.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:30.842 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:30.890 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:30.890 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:30.953 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:30.954+0000 7f95b0813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:30.954 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:30.955+0000 7f95b0813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:30.980 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:30Z"} 2024-03-04T18:20:30.980 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:30Z"} 2024-03-04T18:20:30.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:30.980 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:30.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:30.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T18:20:35.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:35.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:35.982 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:36.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:36.029 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:36.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:36.094+0000 7f1492330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:36.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:36.095+0000 7f1492330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:36.120 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:36Z"} 2024-03-04T18:20:36.120 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:36Z"} 2024-03-04T18:20:36.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:36.121 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:36.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:36.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T18:20:41.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:41.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:41.122 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:41.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:41.231+0000 7fa72612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:41.232 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:41.232+0000 7fa72612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:41.256 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:41Z"} 2024-03-04T18:20:41.257 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:41Z"} 2024-03-04T18:20:41.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:41.257 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:41.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:41.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:20:46.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:46.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:46.258 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:46.306 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:46.306 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:46.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:46.374+0000 7f72b7f0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:46.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:46.375+0000 7f72b7f0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:46.405 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:46Z"} 2024-03-04T18:20:46.405 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:46Z"} 2024-03-04T18:20:46.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:46.406 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:46.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:46.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.608330_559.1 2024-03-04T18:20:51.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:51.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:51.408 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:51.506+0000 7fcea5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:51.507+0000 7fcea5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:51.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:51Z"} 2024-03-04T18:20:51.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:51Z"} 2024-03-04T18:20:51.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:51.530 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:51.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:51.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.881246_635.1 2024-03-04T18:20:56.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:56.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:20:56.531 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:20:56.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:20:56.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:20:56.635 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:56.635+0000 7f08be49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:56.635 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:20:56.636+0000 7f08be49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:20:56.658 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:56Z"} 2024-03-04T18:20:56.658 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:56Z"} 2024-03-04T18:20:56.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:20:56.659 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:56.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:20:56.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:21:01.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:01.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:01.661 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:01.710 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:01.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:01.761 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:01.761+0000 7f93e47a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:01.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:01.762+0000 7f93e47a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:01.784 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:01Z"} 2024-03-04T18:21:01.784 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:01Z"} 2024-03-04T18:21:01.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:01.785 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:01.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:01.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.616944_562.1 2024-03-04T18:21:06.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:06.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:06.787 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:06.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:06.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:06.903 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:06.904+0000 7f4ede667800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:06.904 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:06.905+0000 7f4ede667800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:06.927 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:06Z"} 2024-03-04T18:21:06.928 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:06Z"} 2024-03-04T18:21:06.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:06.928 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:06.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:06.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T18:21:11.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:11.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:11.930 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:12.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:12.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:12.136 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:12.136+0000 7faed4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:12.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:12.139+0000 7faed4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:12.185 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:12Z"} 2024-03-04T18:21:12.185 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:12Z"} 2024-03-04T18:21:12.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:12.185 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:12.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:12.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T18:21:17.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:17.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:17.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:17.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:17.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:17.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:17.300+0000 7f7f84e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:17.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:17.300+0000 7f7f84e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:17.325 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:17Z"} 2024-03-04T18:21:17.326 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:17Z"} 2024-03-04T18:21:17.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:17.326 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:17.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:17.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.641846_565.1 2024-03-04T18:21:22.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:22.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:22.328 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:22.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:22.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:22.436 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:22.437+0000 7fcd9bb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:22.437 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:22.438+0000 7fcd9bb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:22.461 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:22Z"} 2024-03-04T18:21:22.461 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:22Z"} 2024-03-04T18:21:22.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:22.461 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:22.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:22.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.998361_705.1 2024-03-04T18:21:27.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:27.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:27.464 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:27.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:27.561+0000 7f7adf137800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:27.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:27.562+0000 7f7adf137800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:27.591 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:27Z"} 2024-03-04T18:21:27.591 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:27Z"} 2024-03-04T18:21:27.592 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:27.592 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:27.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:27.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.885240_641.1 2024-03-04T18:21:32.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:32.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:32.594 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:32.650 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:32.650 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:32.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:32.712+0000 7f236b5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:32.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:32.713+0000 7f236b5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:32.734 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:32Z"} 2024-03-04T18:21:32.734 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:32Z"} 2024-03-04T18:21:32.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:32.735 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:32.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:32.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.125073_710.1 2024-03-04T18:21:37.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:37.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:37.737 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:37.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:37.848+0000 7f98fff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:37.849 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:37.849+0000 7f98fff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:37.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:37Z"} 2024-03-04T18:21:37.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:37Z"} 2024-03-04T18:21:37.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:37.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:37.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:37.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:21:42.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:42.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:42.880 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:42.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:42.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:42.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:42.974+0000 7f69bb030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:42.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:42.975+0000 7f69bb030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:42.997 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:42Z"} 2024-03-04T18:21:42.997 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:42Z"} 2024-03-04T18:21:42.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:42.997 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:42.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:42.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T18:21:47.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:47.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:47.999 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:48.049 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:48.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:48.104 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:48.105+0000 7fa24a12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:48.105 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:48.106+0000 7fa24a12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:48.132 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:48Z"} 2024-03-04T18:21:48.132 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:48Z"} 2024-03-04T18:21:48.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:48.132 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:48.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:48.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:21:53.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:53.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:53.134 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:53.180 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:53.180 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:53.229+0000 7f0c77f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:53.230+0000 7f0c77f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:53.255 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:53Z"} 2024-03-04T18:21:53.255 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:53Z"} 2024-03-04T18:21:53.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:53.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:53.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:53.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T18:21:58.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:58.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:21:58.257 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:21:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:21:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:21:58.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:58.354+0000 7fefde29d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:58.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:21:58.355+0000 7fefde29d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:21:58.378 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:58Z"} 2024-03-04T18:21:58.378 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:58Z"} 2024-03-04T18:21:58.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:21:58.378 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:58.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:21:58.378 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T18:22:03.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:03.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:03.380 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:03.479+0000 7fd0f4616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:03.479+0000 7fd0f4616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:03.507 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:03Z"} 2024-03-04T18:22:03.507 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:03Z"} 2024-03-04T18:22:03.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:03.507 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:03.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:03.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:22:08.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:08.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:08.509 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:08.562 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:08.563 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:08.633 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:08.633+0000 7fd4b2a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:08.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:08.634+0000 7fd4b2a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:08.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:08Z"} 2024-03-04T18:22:08.667 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:08Z"} 2024-03-04T18:22:08.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:08.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:08.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:08.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T18:22:13.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:13.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:13.669 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:13.723 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:13.723 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:13.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:13.786+0000 7f551908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:13.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:13.787+0000 7f551908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:13.811 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:13Z"} 2024-03-04T18:22:13.812 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:13Z"} 2024-03-04T18:22:13.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:13.812 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:13.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:13.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:22:18.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:18.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:18.813 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:18.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:18.867 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:18.923 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:18.924+0000 7fca66811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:18.924 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:18.925+0000 7fca66811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:18.949 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:18Z"} 2024-03-04T18:22:18.949 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:18Z"} 2024-03-04T18:22:18.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:18.949 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:18.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:18.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:22:23.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:23.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:23.952 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:24.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:24.011 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:24.198 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:24.199+0000 7f5d3232f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:24.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:24.200+0000 7f5d3232f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:24.232 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:24Z"} 2024-03-04T18:22:24.232 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:24Z"} 2024-03-04T18:22:24.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:24.233 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:24.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:24.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.684679_567.1 2024-03-04T18:22:29.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:29.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:29.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:29.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:29.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:29.352+0000 7f730d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:29.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:29.353+0000 7f730d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:29.379 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:29Z"} 2024-03-04T18:22:29.380 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:29Z"} 2024-03-04T18:22:29.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:29.380 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:29.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:29.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T18:22:34.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:34.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:34.382 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:34.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:34.428 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:34.474 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:34.474+0000 7fdc66e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:34.475 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:34.475+0000 7fdc66e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:34.498 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:34Z"} 2024-03-04T18:22:34.499 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:34Z"} 2024-03-04T18:22:34.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:34.499 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:34.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:34.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.828757_695.1 2024-03-04T18:22:39.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:39.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:39.501 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:39.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:39.548 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:39.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:39.606+0000 7fb109a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:39.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:39.607+0000 7fb109a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:39.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:39Z"} 2024-03-04T18:22:39.631 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:39Z"} 2024-03-04T18:22:39.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:39.631 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:39.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:39.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T18:22:44.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:44.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:44.634 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:44.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:44.681 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:44.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:44.737+0000 7f14bb439800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:44.738 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:44.738+0000 7f14bb439800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:44.760 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:44Z"} 2024-03-04T18:22:44.760 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:44Z"} 2024-03-04T18:22:44.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:44.761 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:44.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:44.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:22:49.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:49.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:49.763 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:49.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:49.809 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:49.863 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:49.861+0000 7f5b05a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:49.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:49.862+0000 7f5b05a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:49.885 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:49Z"} 2024-03-04T18:22:49.885 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:49Z"} 2024-03-04T18:22:49.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:49.886 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:49.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:49.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.828757_695.1 2024-03-04T18:22:54.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:54.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:22:54.888 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:22:54.936 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:22:54.936 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:22:55.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:55.007+0000 7fb66d68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:55.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:22:55.008+0000 7fb66d68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:22:55.044 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:55Z"} 2024-03-04T18:22:55.044 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:55Z"} 2024-03-04T18:22:55.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:22:55.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:55.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:22:55.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.890340_643.1 2024-03-04T18:23:00.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:00.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:00.047 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:00.094 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:00.095 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:00.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:00.164+0000 7f1f21230800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:00.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:00.165+0000 7f1f21230800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:00.199 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:00Z"} 2024-03-04T18:23:00.200 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:00Z"} 2024-03-04T18:23:00.200 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:00.200 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:00.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:00.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:23:05.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:05.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:05.202 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:05.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:05.248 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:05.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:05.304+0000 7f233923a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:05.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:05.304+0000 7f233923a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:05.327 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:05Z"} 2024-03-04T18:23:05.327 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:05Z"} 2024-03-04T18:23:05.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:05.327 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:05.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:05.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T18:23:10.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:10.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:10.329 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:10.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:10.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:10.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:10.430+0000 7f89f4130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:10.430 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:10.431+0000 7f89f4130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:10.454 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:10Z"} 2024-03-04T18:23:10.454 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:10Z"} 2024-03-04T18:23:10.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:10.454 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:10.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:10.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T18:23:15.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:15.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:15.456 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:15.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:15.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:15.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:15.572+0000 7fcfec52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:15.572 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:15.573+0000 7fcfec52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:15.596 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:15Z"} 2024-03-04T18:23:15.596 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:15Z"} 2024-03-04T18:23:15.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:15.597 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:15.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:15.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:23:20.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:20.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:20.598 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:20.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:20.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:20.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:20.709+0000 7f49775a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:20.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:20.710+0000 7f49775a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:20.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:20Z"} 2024-03-04T18:23:20.742 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:20Z"} 2024-03-04T18:23:20.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:20.742 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:20.743 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:20.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T18:23:25.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:25.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:25.744 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:25.792 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:25.792 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:25.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:25.841+0000 7f82ab528800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:25.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:25.842+0000 7f82ab528800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:25.864 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:25Z"} 2024-03-04T18:23:25.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:25Z"} 2024-03-04T18:23:25.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:25.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:25.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:25.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T18:23:30.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:30.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:30.866 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:30.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:30.919 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:30.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:30.980+0000 7f00285af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:30.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:30.981+0000 7f00285af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:31.005 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:30Z"} 2024-03-04T18:23:31.005 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:30Z"} 2024-03-04T18:23:31.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:31.006 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:31.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:31.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T18:23:36.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:36.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:36.007 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:36.059 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:36.059 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:36.111+0000 7f0bd9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:36.112+0000 7f0bd9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:36.135 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:36Z"} 2024-03-04T18:23:36.136 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:36Z"} 2024-03-04T18:23:36.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:36.136 DEBUG:rgw_multi.tests:log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:36.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:36.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.987908_703.1 2024-03-04T18:23:41.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2024-03-04T18:23:41.140 DEBUG:boto:path=/sbnvju-75/ 2024-03-04T18:23:41.141 DEBUG:boto:auth_path=/sbnvju-75/ 2024-03-04T18:23:41.141 DEBUG:boto:Method: PUT 2024-03-04T18:23:41.141 DEBUG:boto:Path: /sbnvju-75/ 2024-03-04T18:23:41.141 DEBUG:boto:Data: 2024-03-04T18:23:41.141 DEBUG:boto:Headers: {} 2024-03-04T18:23:41.141 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:23:41.141 DEBUG:boto:Port: 8000 2024-03-04T18:23:41.141 DEBUG:boto:Params: {} 2024-03-04T18:23:41.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:23:41.142 DEBUG:boto:Token: None 2024-03-04T18:23:41.142 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:23:41 GMT /sbnvju-75/ 2024-03-04T18:23:41.142 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:Ygm6okJ797HsBifsSfUv8Q4GPMg= 2024-03-04T18:23:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:23:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ygm6okJ797HsBifsSfUv8Q4GPMg=', 'Content-Length': '0'} 2024-03-04T18:23:41.167 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000aaf5d1e53579147-0065e611ad-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:23:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:23:41.167 DEBUG:rgw_multi.tests:created bucket=sbnvju-75 2024-03-04T18:23:41.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:41.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:41.168 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:41.278 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:23:41.300 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709576621.164199_1064.1","last_update":"2024-03-04T18:23:41.164199Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:23:41.300 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:41.300 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:23:41.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:41.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:41.301 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:41.365 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:41.365 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:41.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:41.443+0000 7f9271a1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:41.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:41.444+0000 7f9271a1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:41.473 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:41Z"} 2024-03-04T18:23:41.473 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:41Z"} 2024-03-04T18:23:41.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:41.473 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:41.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:41.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:23:46.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:46.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:46.475 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:46.521 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:46.522 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:46.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:46.571+0000 7fe92832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:46.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:46.572+0000 7fe92832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:46.596 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:46Z"} 2024-03-04T18:23:46.596 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:46Z"} 2024-03-04T18:23:46.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:46.597 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:46.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:46.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:23:51.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:51.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:51.598 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:51.644 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:51.644 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:51.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:51.695+0000 7fdd29016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:51.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:51.695+0000 7fdd29016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:51.718 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:51Z"} 2024-03-04T18:23:51.718 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:51Z"} 2024-03-04T18:23:51.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:51.719 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:51.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:51.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T18:23:56.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:56.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:23:56.720 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:23:56.766 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:23:56.767 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:23:56.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:56.817+0000 7fe9e3613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:56.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:23:56.818+0000 7fe9e3613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:23:56.841 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:56Z"} 2024-03-04T18:23:56.841 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:56Z"} 2024-03-04T18:23:56.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:23:56.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:56.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:23:56.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.002120_706.1 2024-03-04T18:24:01.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:01.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:01.843 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:01.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:01.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:02.047 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:02.048+0000 7f7aa2933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:02.049 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:02.050+0000 7f7aa2933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:02.074 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:02Z"} 2024-03-04T18:24:02.075 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:02Z"} 2024-03-04T18:24:02.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:02.075 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:02.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:02.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T18:24:07.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:07.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:07.076 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:07.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:07.135 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:07.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:07.193+0000 7fd568b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:07.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:07.194+0000 7fd568b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:07.219 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:07Z"} 2024-03-04T18:24:07.220 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:07Z"} 2024-03-04T18:24:07.220 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:07.220 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:07.220 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:07.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T18:24:12.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:12.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:12.222 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:12.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:12.330+0000 7fc203333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:12.331 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:12.331+0000 7fc203333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:12.353 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:12Z"} 2024-03-04T18:24:12.353 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:12Z"} 2024-03-04T18:24:12.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:12.354 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:12.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:12.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:24:17.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:17.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:17.357 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:17.404 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:17.404 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:17.458 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:17.459+0000 7fc5b7b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:17.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:17.460+0000 7fc5b7b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:17.483 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:17Z"} 2024-03-04T18:24:17.484 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:17Z"} 2024-03-04T18:24:17.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:17.484 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:17.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:17.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:24:22.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:22.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:22.486 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:22.542 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:22.542 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:22.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:22.712+0000 7fe0de213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:22.713+0000 7fe0de213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:22.747 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:22Z"} 2024-03-04T18:24:22.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:22Z"} 2024-03-04T18:24:22.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:22.748 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:22.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:22.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T18:24:27.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:27.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:27.750 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:27.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:27.853+0000 7f566aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:27.853 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:27.854+0000 7f566aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:27.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:27Z"} 2024-03-04T18:24:27.877 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:27Z"} 2024-03-04T18:24:27.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:27.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:27.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:27.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:24:32.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:32.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:32.880 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:32.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:32.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:32.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:32.988+0000 7fa61e942800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:32.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:32.988+0000 7fa61e942800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:33.017 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:32Z"} 2024-03-04T18:24:33.017 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:32Z"} 2024-03-04T18:24:33.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:33.017 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:33.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:33.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T18:24:38.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:38.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:38.019 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:38.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:38.068 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:38.119+0000 7f82ed92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:38.120+0000 7f82ed92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:38.144 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:38Z"} 2024-03-04T18:24:38.144 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:38Z"} 2024-03-04T18:24:38.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:38.144 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:38.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:38.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:24:43.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:43.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:43.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:43.226 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:43.227 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:43.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:43.366+0000 7f3506327800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:43.366 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:43.367+0000 7f3506327800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:43.394 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:43Z"} 2024-03-04T18:24:43.394 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:43Z"} 2024-03-04T18:24:43.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:43.394 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:43.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:43.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:24:48.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:48.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:48.396 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:48.443 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:48.443 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:48.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:48.509+0000 7fc2bc331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:48.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:48.509+0000 7fc2bc331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:48.542 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:48Z"} 2024-03-04T18:24:48.542 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:48Z"} 2024-03-04T18:24:48.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:48.543 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:48.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:48.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:24:53.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:53.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:53.545 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:53.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:53.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:53.651 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:53.652+0000 7fb295d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:53.652 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:53.652+0000 7fb295d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:53.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:53Z"} 2024-03-04T18:24:53.681 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:53Z"} 2024-03-04T18:24:53.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:53.682 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:53.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:53.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T18:24:58.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:58.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:24:58.683 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:24:58.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:24:58.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:24:58.781 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:58.782+0000 7fa2acb39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:58.783 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:24:58.784+0000 7fa2acb39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:24:58.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:58Z"} 2024-03-04T18:24:58.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:58Z"} 2024-03-04T18:24:58.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:24:58.809 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:58.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:24:58.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:25:03.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:03.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:03.811 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:03.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:03.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:03.906 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:03.906+0000 7fe4c5baf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:03.907 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:03.907+0000 7fe4c5baf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:03.932 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:03Z"} 2024-03-04T18:25:03.932 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:03Z"} 2024-03-04T18:25:03.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:03.933 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:03.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:03.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T18:25:08.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:08.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:08.935 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:08.981 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:08.981 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:09.038 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:09.038+0000 7fb8c2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:09.038 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:09.039+0000 7fb8c2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:09.068 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:09Z"} 2024-03-04T18:25:09.068 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:09Z"} 2024-03-04T18:25:09.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:09.069 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:09.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:09.069 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.120400_708.1 2024-03-04T18:25:14.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:14.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:14.071 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:14.118 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:14.118 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:14.182 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:14.183+0000 7f2cf062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:14.183 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:14.183+0000 7f2cf062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:14.214 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:14Z"} 2024-03-04T18:25:14.214 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:14Z"} 2024-03-04T18:25:14.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:14.215 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:14.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:14.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.842248_699.1 2024-03-04T18:25:19.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:19.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:19.217 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:19.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:19.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:19.328 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:19.329+0000 7fb4953a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:19.329 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:19.330+0000 7fb4953a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:19.357 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:19Z"} 2024-03-04T18:25:19.357 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:19Z"} 2024-03-04T18:25:19.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:19.358 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:19.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:19.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T18:25:24.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:24.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:24.359 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:24.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:24.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:24.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:24.462+0000 7fbb1e52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:24.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:24.463+0000 7fbb1e52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:24.489 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:24Z"} 2024-03-04T18:25:24.489 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:24Z"} 2024-03-04T18:25:24.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:24.490 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:24.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:24.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T18:25:29.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:29.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:29.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:29.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:29.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:29.608 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:29.609+0000 7feba6733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:29.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:29.610+0000 7feba6733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:29Z"} 2024-03-04T18:25:29.643 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:29Z"} 2024-03-04T18:25:29.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:29.644 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:29.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:29.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T18:25:34.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:34.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:34.646 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:34.693 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:34.693 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:34.741 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:34.741+0000 7f523d613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:34.742+0000 7f523d613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:34.768 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:34Z"} 2024-03-04T18:25:34.769 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:34Z"} 2024-03-04T18:25:34.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:34.769 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:34.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:34.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.684679_567.1 2024-03-04T18:25:39.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:39.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:39.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:39.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:39.824 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:39.884 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:39.885+0000 7fd9c2011800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:39.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:39.885+0000 7fd9c2011800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:39.911 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:39Z"} 2024-03-04T18:25:39.911 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:39Z"} 2024-03-04T18:25:39.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:39.911 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:39.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:39.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:25:44.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:44.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:44.913 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:44.996 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:44.996 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:45.088 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:45.088+0000 7f1492c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:45.089 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:45.089+0000 7f1492c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:45.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:45Z"} 2024-03-04T18:25:45.113 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:45Z"} 2024-03-04T18:25:45.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:45.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:45.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:45.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.684679_567.1 2024-03-04T18:25:50.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:50.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:50.116 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:50.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:50.162 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:50.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:50.218+0000 7f53a8328800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:50.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:50.219+0000 7f53a8328800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:50.244 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:50Z"} 2024-03-04T18:25:50.244 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:50Z"} 2024-03-04T18:25:50.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:50.245 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:50.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:50.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T18:25:55.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:55.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:25:55.247 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:25:55.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:25:55.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:25:55.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:55.354+0000 7f5803030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:55.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:25:55.355+0000 7f5803030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:25:55.378 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:55Z"} 2024-03-04T18:25:55.378 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:55Z"} 2024-03-04T18:25:55.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:25:55.379 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:55.379 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:25:55.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T18:26:00.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:00.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:00.381 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:00.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:00.429 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:00.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:00.493+0000 7f1c246c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:00.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:00.494+0000 7f1c246c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:00.526 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:00Z"} 2024-03-04T18:26:00.526 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:00Z"} 2024-03-04T18:26:00.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:00.526 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:00.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:00.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571357.886755_642.1 2024-03-04T18:26:05.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:05.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:05.530 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:05.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:05.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:05.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:05.638+0000 7f08e1836800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:05.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:05.639+0000 7f08e1836800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:05.660 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:05Z"} 2024-03-04T18:26:05.661 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:05Z"} 2024-03-04T18:26:05.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:05.661 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:05.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:05.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:26:10.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:10.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:10.663 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:10.712 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:10.712 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:10.779 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:10.779+0000 7f836ac83800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:10.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:10.780+0000 7f836ac83800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:10.810 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:10Z"} 2024-03-04T18:26:10.811 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:10Z"} 2024-03-04T18:26:10.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:10.811 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:10.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:10.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:26:15.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:15.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:15.813 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:15.862 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:15.862 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:15.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:15.918+0000 7f3f7bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:15.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:15.919+0000 7f3f7bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:15.946 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:15Z"} 2024-03-04T18:26:15.946 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:15Z"} 2024-03-04T18:26:15.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:15.946 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:15.947 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:15.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571357.881246_635.1 2024-03-04T18:26:20.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:20.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:20.948 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:21.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:21.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:21.071 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:21.072+0000 7fd63b811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:21.072 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:21.073+0000 7fd63b811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:21.095 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:21Z"} 2024-03-04T18:26:21.096 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:21Z"} 2024-03-04T18:26:21.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:21.096 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:21.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:21.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:26:26.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:26.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:26.098 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:26.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:26.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:26.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:26.222+0000 7f4ef6616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:26.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:26.223+0000 7f4ef6616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:26.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:26Z"} 2024-03-04T18:26:26.249 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:26Z"} 2024-03-04T18:26:26.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:26.249 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:26.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:26.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.688202_568.1 2024-03-04T18:26:31.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:31.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:31.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:31.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:31.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:31.359 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:31.359+0000 7ff395921800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:31.359 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:31.360+0000 7ff395921800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:31.387 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:31Z"} 2024-03-04T18:26:31.387 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:31Z"} 2024-03-04T18:26:31.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:31.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:31.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:31.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T18:26:36.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:36.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:36.390 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:36.443 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:36.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:36.498 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:36.498+0000 7f8cd7d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:36.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:36.499+0000 7f8cd7d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:36.522 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:36Z"} 2024-03-04T18:26:36.522 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:36Z"} 2024-03-04T18:26:36.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:36.523 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:36.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:36.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T18:26:41.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:41.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:41.524 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:41.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:41.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:41.726 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:41.727+0000 7f1d97a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:41.727 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:41.728+0000 7f1d97a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:41.764 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:41Z"} 2024-03-04T18:26:41.764 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:41Z"} 2024-03-04T18:26:41.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:41.764 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:41.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:41.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:26:46.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:46.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:46.766 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:46.839 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:46.839 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:46.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:46.939+0000 7f9939b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:46.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:46.940+0000 7f9939b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:46.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:46Z"} 2024-03-04T18:26:46.981 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:46Z"} 2024-03-04T18:26:46.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:46.982 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:46.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:46.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T18:26:51.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:51.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:51.983 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:52.089 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:52.089+0000 7f4c11d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:52.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:52.090+0000 7f4c11d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:52.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:52Z"} 2024-03-04T18:26:52.113 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:52Z"} 2024-03-04T18:26:52.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:52.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:52.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:52.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.613615_561.1 2024-03-04T18:26:57.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:57.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:26:57.114 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:26:57.173 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:26:57.173 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:26:57.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:57.242+0000 7f0f29b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:57.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:26:57.243+0000 7f0f29b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:26:57.273 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:57Z"} 2024-03-04T18:26:57.273 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:57Z"} 2024-03-04T18:26:57.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:26:57.273 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:57.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:26:57.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.640587_564.1 2024-03-04T18:27:02.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:02.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:02.274 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:02.321 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:02.321 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:02.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:02.376+0000 7f1b6cb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:02.375 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:02.377+0000 7f1b6cb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:02.397 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:02Z"} 2024-03-04T18:27:02.397 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:02Z"} 2024-03-04T18:27:02.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:02.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:02.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:02.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T18:27:07.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:07.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:07.399 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:07.500+0000 7fb83c14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:07.501+0000 7fb83c14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:07.526 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:07Z"} 2024-03-04T18:27:07.526 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:07Z"} 2024-03-04T18:27:07.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:07.526 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:07.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:07.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.835525_697.1 2024-03-04T18:27:12.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:12.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:12.528 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:12.574 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:12.574 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:12.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:12.645+0000 7f58033a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:12.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:12.646+0000 7f58033a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:12.672 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:12Z"} 2024-03-04T18:27:12.672 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:12Z"} 2024-03-04T18:27:12.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:12.673 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:12.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:12.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T18:27:17.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:17.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:17.674 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:17.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:17.799+0000 7f57d180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:17.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:17.800+0000 7f57d180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:17.825 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:17Z"} 2024-03-04T18:27:17.825 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:17Z"} 2024-03-04T18:27:17.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:17.826 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:17.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:17.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:27:22.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:22.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:22.827 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:22.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:22.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:22.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:22.935+0000 7fd5f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:22.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:22.936+0000 7fd5f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:22.962 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:22Z"} 2024-03-04T18:27:22.962 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:22Z"} 2024-03-04T18:27:22.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:22.963 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:22.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:22.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T18:27:27.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:27.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:27.964 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:28.014 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:28.014 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:28.083 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:28.084+0000 7f9bcd40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:28.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:28.085+0000 7f9bcd40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:28.111 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:28Z"} 2024-03-04T18:27:28.111 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:28Z"} 2024-03-04T18:27:28.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:28.112 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:28.112 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:28.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T18:27:33.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:33.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:33.115 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:33.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:33.224+0000 7ff339911800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:33.225 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:33.225+0000 7ff339911800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:33.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:33Z"} 2024-03-04T18:27:33.248 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:33Z"} 2024-03-04T18:27:33.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:33.249 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:33.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:33.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:27:38.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:38.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:38.251 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:38.302 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:38.370 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:38.371+0000 7f517e68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:38.371 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:38.372+0000 7f517e68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:38.398 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:38Z"} 2024-03-04T18:27:38.398 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:38Z"} 2024-03-04T18:27:38.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:38.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:38.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:38.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.116657_707.1 2024-03-04T18:27:43.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:43.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:43.400 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:43.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:43.455 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:43.516 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:43.517+0000 7f1de3933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:43.517 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:43.517+0000 7f1de3933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:43.589 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:43Z"} 2024-03-04T18:27:43.590 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:43Z"} 2024-03-04T18:27:43.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:43.590 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:43.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:43.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T18:27:48.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:48.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:48.592 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:48.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:48.648 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:48.717 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:48.718+0000 7fcf850c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:48.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:48.719+0000 7fcf850c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:48.744 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:48Z"} 2024-03-04T18:27:48.745 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:48Z"} 2024-03-04T18:27:48.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:48.746 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:48.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:48.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.644921_566.1 2024-03-04T18:27:53.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:53.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:53.748 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:53.830 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:53.830 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:53.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:53.966+0000 7fa10c93c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:53.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:53.967+0000 7fa10c93c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:53.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:53Z"} 2024-03-04T18:27:53.994 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:53Z"} 2024-03-04T18:27:53.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:53.994 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:53.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:53.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T18:27:58.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:58.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:27:58.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:27:59.079 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:27:59.080 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:27:59.169 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:59.170+0000 7f35b314b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:59.170 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:27:59.171+0000 7f35b314b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:27:59.207 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:59Z"} 2024-03-04T18:27:59.208 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:59Z"} 2024-03-04T18:27:59.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:27:59.208 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:59.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:27:59.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.996460_704.1 2024-03-04T18:28:04.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:04.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:04.210 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:04.418+0000 7f5cd949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:04.419+0000 7f5cd949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:04.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:04Z"} 2024-03-04T18:28:04.442 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:04Z"} 2024-03-04T18:28:04.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:04.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:04.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:04.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T18:28:09.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:09.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:09.445 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:09.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:09.541+0000 7f810bba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:09.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:09.542+0000 7f810bba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:09.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:09Z"} 2024-03-04T18:28:09.564 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:09Z"} 2024-03-04T18:28:09.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:09.565 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:09.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:09.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T18:28:14.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:14.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:14.567 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:14.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:14.614 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:14.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:14.667+0000 7ff30573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:14.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:14.668+0000 7ff30573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:14.698 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:14Z"} 2024-03-04T18:28:14.698 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:14Z"} 2024-03-04T18:28:14.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:14.698 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:14.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:14.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T18:28:19.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:19.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:19.700 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:19.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:19.746 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:19.804 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:19.804+0000 7fd6f8530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:19.804 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:19.805+0000 7fd6f8530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:19.828 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:19Z"} 2024-03-04T18:28:19.831 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:19Z"} 2024-03-04T18:28:19.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:19.831 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:19.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:19.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T18:28:24.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:24.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:24.833 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:24.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:24.881 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:24.942 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:24.943+0000 7f2f8733c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:24.943 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:24.944+0000 7f2f8733c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:24.975 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:24Z"} 2024-03-04T18:28:24.975 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:24Z"} 2024-03-04T18:28:24.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:24.975 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:24.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:24.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T18:28:29.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:29.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:29.977 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:30.023 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:30.023 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:30.076+0000 7fb05e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:30.077+0000 7fb05e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:30.100 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:30Z"} 2024-03-04T18:28:30.100 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:30Z"} 2024-03-04T18:28:30.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:30.100 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:30.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:30.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:28:35.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:35.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:35.102 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:35.148 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:35.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:35.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:35.201+0000 7f4766d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:35.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:35.202+0000 7f4766d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:35.225 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:35Z"} 2024-03-04T18:28:35.225 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:35Z"} 2024-03-04T18:28:35.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:35.225 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:35.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:35.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.613615_561.1 2024-03-04T18:28:40.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:40.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:40.227 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:40.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:40.280 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:40.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:40.333+0000 7f613ac20800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:40.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:40.334+0000 7f613ac20800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:40.357 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:40Z"} 2024-03-04T18:28:40.357 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:40Z"} 2024-03-04T18:28:40.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:40.357 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:40.357 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:40.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T18:28:45.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:45.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:45.359 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:45.405 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:45.406 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:45.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:45.454+0000 7f1f3ddaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:45.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:45.455+0000 7f1f3ddaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:45.477 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:45Z"} 2024-03-04T18:28:45.477 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:45Z"} 2024-03-04T18:28:45.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:45.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:45.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:45.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T18:28:50.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-03-04T18:28:50.482 DEBUG:boto:path=/sbnvju-76/ 2024-03-04T18:28:50.482 DEBUG:boto:auth_path=/sbnvju-76/ 2024-03-04T18:28:50.482 DEBUG:boto:Method: PUT 2024-03-04T18:28:50.482 DEBUG:boto:Path: /sbnvju-76/ 2024-03-04T18:28:50.482 DEBUG:boto:Data: 2024-03-04T18:28:50.482 DEBUG:boto:Headers: {} 2024-03-04T18:28:50.482 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:28:50.483 DEBUG:boto:Port: 8000 2024-03-04T18:28:50.483 DEBUG:boto:Params: {} 2024-03-04T18:28:50.483 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:28:50.483 DEBUG:boto:Token: None 2024-03-04T18:28:50.483 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:28:50 GMT /sbnvju-76/ 2024-03-04T18:28:50.484 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:TS+OuCdPzCwRhJi0qmb2yzldwH8= 2024-03-04T18:28:50.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:28:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TS+OuCdPzCwRhJi0qmb2yzldwH8=', 'Content-Length': '0'} 2024-03-04T18:28:50.508 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000450192d64099413c-0065e612e2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:28:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:28:50.509 DEBUG:rgw_multi.tests:created bucket=sbnvju-76 2024-03-04T18:28:50.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:50.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:50.509 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:50.564 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:50.564 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:28:50.643 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709576930.507437_1081.1","last_update":"2024-03-04T18:28:50.507437Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:28:50.643 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:50.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:28:50.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:50.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:50.644 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:50.713 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:50.713 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:50.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:50.788+0000 7f0ba6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:50.789 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:50.789+0000 7f0ba6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:50.817 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:50Z"} 2024-03-04T18:28:50.817 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:50Z"} 2024-03-04T18:28:50.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:50.818 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:50.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:50.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T18:28:55.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:55.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:28:55.819 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:28:55.866 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:28:55.866 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:28:56.041 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:56.041+0000 7ff2e5d27800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:56.043 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:28:56.043+0000 7ff2e5d27800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:28:56.098 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:56Z"} 2024-03-04T18:28:56.098 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:56Z"} 2024-03-04T18:28:56.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:28:56.098 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:56.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:28:56.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T18:29:01.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:01.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:01.100 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:01.160 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:01.160 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:01.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:01.296+0000 7fa555c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:01.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:01.298+0000 7fa555c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:01.337 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:01Z"} 2024-03-04T18:29:01.337 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:01Z"} 2024-03-04T18:29:01.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:01.337 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:01.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:01.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.881246_635.1 2024-03-04T18:29:06.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:06.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:06.339 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:06.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:06.390 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:06.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:06.443+0000 7ff553d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:06.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:06.444+0000 7ff553d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:06.471 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:06Z"} 2024-03-04T18:29:06.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:06Z"} 2024-03-04T18:29:06.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:06.471 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:06.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:06.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.125073_710.1 2024-03-04T18:29:11.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:11.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:11.473 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:11.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:11.523 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:11.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:11.574+0000 7f2b0013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:11.575 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:11.575+0000 7f2b0013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:11.598 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:11Z"} 2024-03-04T18:29:11.598 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:11Z"} 2024-03-04T18:29:11.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:11.598 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:11.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:11.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:29:16.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:16.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:16.600 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:16.652 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:16.652 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:16.716 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:16.717+0000 7f4fd4222800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:16.717 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:16.718+0000 7f4fd4222800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:16.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:16Z"} 2024-03-04T18:29:16.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:16Z"} 2024-03-04T18:29:16.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:16.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:16.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:16.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.637258_563.1 2024-03-04T18:29:21.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:21.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:21.749 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:21.799 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:21.800 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:21.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:21.865+0000 7f50ab610800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:21.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:21.866+0000 7f50ab610800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:21.888 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:21Z"} 2024-03-04T18:29:21.889 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:21Z"} 2024-03-04T18:29:21.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:21.889 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:21.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:21.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.998361_705.1 2024-03-04T18:29:26.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:26.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:26.891 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:26.944 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:26.945 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:27.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:27.029+0000 7f98525a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:27.029 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:27.030+0000 7f98525a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:27.066 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:27Z"} 2024-03-04T18:29:27.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:27Z"} 2024-03-04T18:29:27.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:27.067 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:27.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:27.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:29:32.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:32.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:32.069 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:32.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:32.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:32.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:32.308+0000 7f4c1c1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:32.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:32.308+0000 7f4c1c1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:32.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:32Z"} 2024-03-04T18:29:32.330 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:32Z"} 2024-03-04T18:29:32.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:32.331 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:32.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:32.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.890340_643.1 2024-03-04T18:29:37.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:37.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:37.333 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:37.394 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:37.394 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:37.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:37.463+0000 7fdfe4330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:37.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:37.464+0000 7fdfe4330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:37.486 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:37Z"} 2024-03-04T18:29:37.487 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:37Z"} 2024-03-04T18:29:37.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:37.487 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:37.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:37.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.613615_561.1 2024-03-04T18:29:42.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:42.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:42.489 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:42.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:42.561 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:42.633 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:42.634+0000 7ff271330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:42.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:42.635+0000 7ff271330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:42.665 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:42Z"} 2024-03-04T18:29:42.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:42Z"} 2024-03-04T18:29:42.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:42.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:42.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:42.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T18:29:47.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:47.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:47.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:47.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:47.740 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:47.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:47.804+0000 7f5bedd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:47.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:47.805+0000 7f5bedd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:47.830 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:47Z"} 2024-03-04T18:29:47.830 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:47Z"} 2024-03-04T18:29:47.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:47.831 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:47.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:47.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:29:52.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:52.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:52.833 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:52.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:52.896 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:52.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:52.958+0000 7fd972d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:52.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:52.959+0000 7fd972d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:52.994 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:52Z"} 2024-03-04T18:29:52.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:52Z"} 2024-03-04T18:29:52.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:52.995 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:52.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:52.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:29:57.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:57.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:29:57.996 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:29:58.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:29:58.043 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:29:58.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:58.096+0000 7fc15f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:58.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:29:58.097+0000 7fc15f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:29:58.119 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:58Z"} 2024-03-04T18:29:58.120 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:58Z"} 2024-03-04T18:29:58.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:29:58.120 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:58.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:29:58.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:30:03.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:03.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:03.122 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:03.218+0000 7f7f98616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:03.219+0000 7f7f98616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:03.241 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:03Z"} 2024-03-04T18:30:03.241 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:03Z"} 2024-03-04T18:30:03.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:03.241 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:03.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:03.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T18:30:08.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:08.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:08.243 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:08.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:08.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:08.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:08.351+0000 7f0e48c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:08.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:08.352+0000 7f0e48c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:08.375 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:08Z"} 2024-03-04T18:30:08.375 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:08Z"} 2024-03-04T18:30:08.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:08.376 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:08.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:08.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T18:30:13.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:13.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:13.377 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:13.424 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:13.424 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:13.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:13.473+0000 7fc2e4c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:13.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:13.474+0000 7fc2e4c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:13.500 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:13Z"} 2024-03-04T18:30:13.501 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:13Z"} 2024-03-04T18:30:13.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:13.501 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:13.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:13.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.886755_642.1 2024-03-04T18:30:18.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:18.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:18.503 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:18.549 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:18.550 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:18.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:18.602+0000 7fdf0949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:18.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:18.603+0000 7fdf0949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:18.625 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:18Z"} 2024-03-04T18:30:18.626 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:18Z"} 2024-03-04T18:30:18.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:18.626 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:18.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:18.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:30:23.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:23.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:23.628 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:23.673 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:23.673 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:23.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:23.721+0000 7f16aef2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:23.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:23.722+0000 7f16aef2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:23.743 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:23Z"} 2024-03-04T18:30:23.743 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:23Z"} 2024-03-04T18:30:23.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:23.744 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:23.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:23.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.175444_719.1 2024-03-04T18:30:28.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:28.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:28.745 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:28.792 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:28.793 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:28.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:28.843+0000 7f4bdf135800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:28.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:28.844+0000 7f4bdf135800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:28Z"} 2024-03-04T18:30:28.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:28Z"} 2024-03-04T18:30:28.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:28.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:28.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:28.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T18:30:33.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:33.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:33.868 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:33.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:33.915 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:33.979 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:33.980+0000 7f8e2f330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:33.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:33.981+0000 7f8e2f330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:34.006 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:33Z"} 2024-03-04T18:30:34.006 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:33Z"} 2024-03-04T18:30:34.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:34.006 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:34.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:34.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T18:30:39.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:39.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:39.008 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:39.055 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:39.055 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:39.115 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:39.116+0000 7f886d72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:39.116 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:39.117+0000 7f886d72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:39.138 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:39Z"} 2024-03-04T18:30:39.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:39Z"} 2024-03-04T18:30:39.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:39.139 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:39.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:39.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:30:44.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:44.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:44.141 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:44.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:44.187 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:44.242+0000 7f82df013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:44.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:44.243+0000 7f82df013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:44.273 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:44Z"} 2024-03-04T18:30:44.273 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:44Z"} 2024-03-04T18:30:44.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:44.274 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:44.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:44.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T18:30:49.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:49.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:49.275 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:49.322 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:49.322 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:49.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:49.369+0000 7fd663d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:49.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:49.370+0000 7fd663d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:49.392 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:49Z"} 2024-03-04T18:30:49.392 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:49Z"} 2024-03-04T18:30:49.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:49.392 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:49.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:49.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T18:30:54.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:54.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:54.394 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:54.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:54.442 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:54.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:54.502+0000 7f3d9e521800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:54.503 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:54.503+0000 7f3d9e521800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:54.536 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:54Z"} 2024-03-04T18:30:54.536 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:54Z"} 2024-03-04T18:30:54.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:54.536 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:54.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:54.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:30:59.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:59.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:30:59.538 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:30:59.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:30:59.591 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:30:59.656 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:59.657+0000 7fe0a794b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:59.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:30:59.658+0000 7fe0a794b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:30:59.682 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:59Z"} 2024-03-04T18:30:59.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:59Z"} 2024-03-04T18:30:59.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:30:59.682 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:59.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:30:59.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T18:31:04.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:04.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:04.684 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:04.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:04.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:04.798+0000 7ff2f2230800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:04.798 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:04.799+0000 7ff2f2230800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:04.827 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:04Z"} 2024-03-04T18:31:04.827 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:04Z"} 2024-03-04T18:31:04.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:04.827 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:04.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:04.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T18:31:09.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:09.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:09.829 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:09.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:09.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:09.933 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:09.933+0000 7f8c51d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:09.934 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:09.934+0000 7f8c51d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:09.956 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:09Z"} 2024-03-04T18:31:09.957 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:09Z"} 2024-03-04T18:31:09.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:09.957 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:09.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:09.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:31:14.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:14.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:14.959 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:15.013 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:15.014 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:15.074+0000 7f82d9f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:15.075 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:15.075+0000 7f82d9f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:15.098 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:15Z"} 2024-03-04T18:31:15.098 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:15Z"} 2024-03-04T18:31:15.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:15.099 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:15.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:15.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.616944_562.1 2024-03-04T18:31:20.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:20.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:20.100 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:20.154 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:20.154 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:20.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:20.228+0000 7facd4e36800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:20.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:20.229+0000 7facd4e36800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:20.255 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:20Z"} 2024-03-04T18:31:20.256 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:20Z"} 2024-03-04T18:31:20.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:20.256 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:20.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:20.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T18:31:25.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:25.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:25.258 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:25.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:25.364+0000 7fe404c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:25.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:25.365+0000 7fe404c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:25.391 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:25Z"} 2024-03-04T18:31:25.391 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:25Z"} 2024-03-04T18:31:25.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:25.392 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:25.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:25.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.637258_563.1 2024-03-04T18:31:30.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:30.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:30.393 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:30.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:30.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:30.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:30.505+0000 7fea1db31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:30.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:30.506+0000 7fea1db31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:30.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:30Z"} 2024-03-04T18:31:30.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:30Z"} 2024-03-04T18:31:30.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:30.538 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:30.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:30.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T18:31:35.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:35.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:35.540 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:35.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:35.679+0000 7f7923f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:35.681+0000 7f7923f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:35.784 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:35Z"} 2024-03-04T18:31:35.784 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:35Z"} 2024-03-04T18:31:35.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:35.785 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:35.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:35.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:31:40.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:40.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:40.787 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:40.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:40.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:40.894+0000 7f7dbdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:40.896+0000 7f7dbdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:40.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:40Z"} 2024-03-04T18:31:40.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:40Z"} 2024-03-04T18:31:40.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:40.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:40.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:40.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.689443_569.1 2024-03-04T18:31:45.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:45.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:45.926 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:45.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:45.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:46.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:46.060+0000 7f906231f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:46.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:46.061+0000 7f906231f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:46.094 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:46Z"} 2024-03-04T18:31:46.094 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:46Z"} 2024-03-04T18:31:46.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:46.094 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:46.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:46.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:31:51.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:51.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:51.096 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:51.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:51.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:51.208+0000 7f7846b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:51.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:51.209+0000 7f7846b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:51.232 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:51Z"} 2024-03-04T18:31:51.232 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:51Z"} 2024-03-04T18:31:51.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:51.232 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:51.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:51.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T18:31:56.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:56.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:31:56.234 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:31:56.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:31:56.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:31:56.354 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:56.355+0000 7ff3e8133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:56.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:31:56.356+0000 7ff3e8133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:31:56.385 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:56Z"} 2024-03-04T18:31:56.385 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:56Z"} 2024-03-04T18:31:56.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:31:56.385 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:56.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:31:56.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:32:01.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:01.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:01.387 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:01.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:01.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:01.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:01.495+0000 7f7280724800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:01.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:01.496+0000 7f7280724800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:01.528 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:01Z"} 2024-03-04T18:32:01.528 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:01Z"} 2024-03-04T18:32:01.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:01.529 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:01.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:01.529 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.885240_641.1 2024-03-04T18:32:06.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:06.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:06.530 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:06.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:06.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:06.640 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:06.641+0000 7f8471130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:06.641 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:06.642+0000 7f8471130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:06.665 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:06Z"} 2024-03-04T18:32:06.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:06Z"} 2024-03-04T18:32:06.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:06.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:06.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:06.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.612016_560.1 2024-03-04T18:32:11.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:11.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:11.667 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:11.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:11.715 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:11.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:11.780+0000 7fd281f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:11.781 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:11.781+0000 7fd281f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:11.814 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:11Z"} 2024-03-04T18:32:11.814 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:11Z"} 2024-03-04T18:32:11.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:11.815 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:11.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:11.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T18:32:16.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:16.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:16.816 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:16.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:16.863 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:16.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:16.930+0000 7fba84039800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:16.930 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:16.931+0000 7fba84039800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:16.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:16Z"} 2024-03-04T18:32:16.959 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:16Z"} 2024-03-04T18:32:16.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:16.959 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:16.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:16.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T18:32:21.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:21.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:21.961 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:22.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:22.009 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:22.076+0000 7f1f6048a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:22.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:22.077+0000 7f1f6048a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:22.108 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:22Z"} 2024-03-04T18:32:22.108 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:22Z"} 2024-03-04T18:32:22.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:22.108 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:22.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:22.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T18:32:27.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:27.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:27.110 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:27.157 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:27.157 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:27.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:27.212+0000 7f1bb4624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:27.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:27.213+0000 7f1bb4624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:27.236 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:27Z"} 2024-03-04T18:32:27.236 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:27Z"} 2024-03-04T18:32:27.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:27.237 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:27.237 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:27.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.828757_695.1 2024-03-04T18:32:32.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:32.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:32.238 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:32.286 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:32.286 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:32.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:32.340+0000 7f8782331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:32.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:32.341+0000 7f8782331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:32.366 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:32Z"} 2024-03-04T18:32:32.367 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:32Z"} 2024-03-04T18:32:32.367 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:32.367 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:32.367 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:32.367 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.842248_699.1 2024-03-04T18:32:37.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:37.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:37.369 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:37.422 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:37.422 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:37.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:37.497+0000 7f833e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:37.497 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:37.498+0000 7f833e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:37.571 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:37Z"} 2024-03-04T18:32:37.571 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:37Z"} 2024-03-04T18:32:37.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:37.572 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:37.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:37.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.881246_635.1 2024-03-04T18:32:42.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:42.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:42.574 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:42.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:42.645 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:42.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:42.722+0000 7faa4fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:42.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:42.723+0000 7faa4fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:42.757 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:42Z"} 2024-03-04T18:32:42.757 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:42Z"} 2024-03-04T18:32:42.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:42.758 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:42.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:42.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:32:47.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:47.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:47.760 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:47.834 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:47.834 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:47.922 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:47.922+0000 7f9d73210800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:47.922 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:47.923+0000 7f9d73210800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:47.954 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:47Z"} 2024-03-04T18:32:47.955 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:47Z"} 2024-03-04T18:32:47.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:47.955 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:47.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:47.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.154969_713.1 2024-03-04T18:32:52.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:52.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:52.957 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:53.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:53.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:53.069 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:53.069+0000 7fc8d612b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:53.069 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:53.070+0000 7fc8d612b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:53.095 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:53Z"} 2024-03-04T18:32:53.095 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:53Z"} 2024-03-04T18:32:53.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:53.096 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:53.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:53.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.153603_712.1 2024-03-04T18:32:58.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:58.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:32:58.097 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:32:58.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:32:58.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:32:58.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:58.205+0000 7fe8e1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:58.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:32:58.206+0000 7fe8e1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:32:58.233 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:58Z"} 2024-03-04T18:32:58.233 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:58Z"} 2024-03-04T18:32:58.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:32:58.234 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:58.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:32:58.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T18:33:03.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:03.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:03.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:03.340+0000 7efdad013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:03.341+0000 7efdad013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:03.363 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:03Z"} 2024-03-04T18:33:03.363 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:03Z"} 2024-03-04T18:33:03.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:03.363 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:03.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:03.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.641846_565.1 2024-03-04T18:33:08.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:08.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:08.365 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:08.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:08.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:08.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:08.469+0000 7efe4ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:08.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:08.469+0000 7efe4ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:08.497 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:08Z"} 2024-03-04T18:33:08.497 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:08Z"} 2024-03-04T18:33:08.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:08.497 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:08.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:08.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.116657_707.1 2024-03-04T18:33:13.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:13.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:13.501 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:13.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:13.555 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:13.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:13.615+0000 7ff1d031e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:13.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:13.616+0000 7ff1d031e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:13.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:13Z"} 2024-03-04T18:33:13.645 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:13Z"} 2024-03-04T18:33:13.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:13.646 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:13.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:13.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.002120_706.1 2024-03-04T18:33:18.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:18.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:18.648 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:18.698 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:18.698 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:18.752 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:18.753+0000 7f7dd192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:18.753 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:18.754+0000 7f7dd192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:18.776 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:18Z"} 2024-03-04T18:33:18.776 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:18Z"} 2024-03-04T18:33:18.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:18.776 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:18.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:18.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T18:33:23.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:23.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:23.778 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:23.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:23.828 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:23.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:23.880+0000 7fd78bc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:23.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:23.881+0000 7fd78bc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:23.904 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:23Z"} 2024-03-04T18:33:23.904 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:23Z"} 2024-03-04T18:33:23.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:23.904 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:23.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:23.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T18:33:28.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:28.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:28.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:28.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:28.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:29.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:29.017+0000 7fa082c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:29.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:29.018+0000 7fa082c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:29.041 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:29Z"} 2024-03-04T18:33:29.041 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:29Z"} 2024-03-04T18:33:29.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:29.041 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:29.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:29.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.885240_641.1 2024-03-04T18:33:34.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:34.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:34.044 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:34.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:34.214+0000 7f9adc62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:34.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:34.217+0000 7f9adc62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:34.290 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:34Z"} 2024-03-04T18:33:34.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:34Z"} 2024-03-04T18:33:34.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:34.291 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:34.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:34.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T18:33:39.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:39.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:39.293 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:39.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:39.412+0000 7fbe89a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:39.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:39.413+0000 7fbe89a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:39.443 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:39Z"} 2024-03-04T18:33:39.443 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:39Z"} 2024-03-04T18:33:39.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:39.444 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:39.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:39.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T18:33:44.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:44.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:44.446 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:44.521 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:44.521 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:44.628 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:44.628+0000 7fbf47817800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:44.628 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:44.629+0000 7fbf47817800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:44.654 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:44Z"} 2024-03-04T18:33:44.654 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:44Z"} 2024-03-04T18:33:44.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:44.655 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:44.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:44.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T18:33:49.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:49.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:49.656 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:49.718 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:49.719 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:49.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:49.767+0000 7fcafa526800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:49.768 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:49.768+0000 7fcafa526800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:49.790 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:49Z"} 2024-03-04T18:33:49.790 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:49Z"} 2024-03-04T18:33:49.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:49.790 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:49.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:49.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.641846_565.1 2024-03-04T18:33:54.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:54.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:54.792 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:33:54.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:33:54.854 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:33:54.912 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:54.913+0000 7f1a00d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:54.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:33:54.914+0000 7f1a00d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:33:54.940 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:54Z"} 2024-03-04T18:33:54.941 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:54Z"} 2024-03-04T18:33:54.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:33:54.941 DEBUG:rgw_multi.tests:log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:54.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:33:54.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T18:33:59.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-03-04T18:33:59.946 DEBUG:boto:path=/sbnvju-77/ 2024-03-04T18:33:59.946 DEBUG:boto:auth_path=/sbnvju-77/ 2024-03-04T18:33:59.946 DEBUG:boto:Method: PUT 2024-03-04T18:33:59.946 DEBUG:boto:Path: /sbnvju-77/ 2024-03-04T18:33:59.946 DEBUG:boto:Data: 2024-03-04T18:33:59.946 DEBUG:boto:Headers: {} 2024-03-04T18:33:59.946 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:33:59.946 DEBUG:boto:Port: 8000 2024-03-04T18:33:59.947 DEBUG:boto:Params: {} 2024-03-04T18:33:59.947 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:33:59.947 DEBUG:boto:Token: None 2024-03-04T18:33:59.947 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:33:59 GMT /sbnvju-77/ 2024-03-04T18:33:59.948 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:GvzBw6WI6KoQuRHRFoKcRvYGrBs= 2024-03-04T18:33:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GvzBw6WI6KoQuRHRFoKcRvYGrBs=', 'Content-Length': '0'} 2024-03-04T18:33:59.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000012d8399997fa87d9-0065e61417-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:33:59.971 DEBUG:rgw_multi.tests:created bucket=sbnvju-77 2024-03-04T18:33:59.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:59.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:33:59.972 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:34:00.098 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577239.970066_1090.1","last_update":"2024-03-04T18:33:59.970066Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:34:00.099 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:00.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:34:00.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:00.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:00.099 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:00.164 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:00.165 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:00.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:00.298+0000 7f7266d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:00.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:00.299+0000 7f7266d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:00.329 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:00Z"} 2024-03-04T18:34:00.329 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:00Z"} 2024-03-04T18:34:00.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:00.330 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:00.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:00.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:34:05.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:05.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:05.332 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:05.377 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:05.378 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:05.427 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:05.427+0000 7fb407015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:05.428 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:05.428+0000 7fb407015800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:05.452 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:05Z"} 2024-03-04T18:34:05.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:05Z"} 2024-03-04T18:34:05.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:05.453 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:05.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:05.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T18:34:10.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:10.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:10.455 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:10.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:10.503 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:10.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:10.563+0000 7fe9df444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:10.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:10.564+0000 7fe9df444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:10.589 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:10Z"} 2024-03-04T18:34:10.589 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:10Z"} 2024-03-04T18:34:10.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:10.589 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:10.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:10.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:34:15.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:15.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:15.591 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:15.640 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:15.640 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:15.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:15.692+0000 7fb10af2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:15.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:15.692+0000 7fb10af2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:15.722 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:15Z"} 2024-03-04T18:34:15.722 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:15Z"} 2024-03-04T18:34:15.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:15.722 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:15.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:15.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.641846_565.1 2024-03-04T18:34:20.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:20.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:20.724 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:20.777 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:20.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:20.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:20.840+0000 7f5eb6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:20.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:20.841+0000 7f5eb6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:20.865 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:20Z"} 2024-03-04T18:34:20.865 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:20Z"} 2024-03-04T18:34:20.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:20.865 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:20.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:20.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T18:34:25.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:25.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:25.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:25.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:25.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:25.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:25.981+0000 7f8c6f530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:25.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:25.982+0000 7f8c6f530800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:26.010 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:25Z"} 2024-03-04T18:34:26.010 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:25Z"} 2024-03-04T18:34:26.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:26.011 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:26.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:26.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:34:31.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:31.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:31.012 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:31.066 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:31.066 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:31.128 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:31.129+0000 7f58ba12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:31.129 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:31.130+0000 7f58ba12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:31.157 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:31Z"} 2024-03-04T18:34:31.158 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:31Z"} 2024-03-04T18:34:31.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:31.158 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:31.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:31.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.641846_565.1 2024-03-04T18:34:36.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:36.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:36.159 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:36.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:36.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:36.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:36.270+0000 7f829ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:36.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:36.271+0000 7f829ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:36.342 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:36Z"} 2024-03-04T18:34:36.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:36Z"} 2024-03-04T18:34:36.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:36.343 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:36.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:36.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T18:34:41.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:41.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:41.344 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:41.432 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:41.432 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:41.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:41.555+0000 7efd7e62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:41.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:41.556+0000 7efd7e62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:41.585 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:41Z"} 2024-03-04T18:34:41.585 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:41Z"} 2024-03-04T18:34:41.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:41.585 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:41.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:41.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.692350_570.1 2024-03-04T18:34:46.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:46.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:46.587 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:46.633 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:46.633 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:46.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:46.694+0000 7f2b6d327800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:46.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:46.695+0000 7f2b6d327800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:46.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:46Z"} 2024-03-04T18:34:46.720 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:46Z"} 2024-03-04T18:34:46.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:46.720 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:46.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:46.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:34:51.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:51.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:51.722 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:51.769 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:51.769 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:51.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:51.842+0000 7f876534b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:51.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:51.843+0000 7f876534b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:51.869 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:51Z"} 2024-03-04T18:34:51.869 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:51Z"} 2024-03-04T18:34:51.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:51.870 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:51.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:51.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T18:34:56.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:56.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:34:56.871 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:34:56.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:34:56.924 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:34:56.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:56.999+0000 7f644bc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:56.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:34:57.000+0000 7f644bc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:34:57.032 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:57Z"} 2024-03-04T18:34:57.033 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:57Z"} 2024-03-04T18:34:57.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:34:57.033 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:57.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:34:57.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:35:02.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:02.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:02.034 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:02.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:02.084 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:02.134 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:02.135+0000 7f3d21412800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:02.136+0000 7f3d21412800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:02.157 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:02Z"} 2024-03-04T18:35:02.158 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:02Z"} 2024-03-04T18:35:02.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:02.158 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:02.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:02.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.890340_643.1 2024-03-04T18:35:07.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:07.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:07.163 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:07.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:07.271+0000 7f1060a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:07.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:07.272+0000 7f1060a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:07.304 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:07Z"} 2024-03-04T18:35:07.304 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:07Z"} 2024-03-04T18:35:07.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:07.304 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:07.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:07.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.120400_708.1 2024-03-04T18:35:12.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:12.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:12.306 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:12.359 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:12.359 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:12.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:12.421+0000 7fe461f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:12.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:12.422+0000 7fe461f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:12.446 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:12Z"} 2024-03-04T18:35:12.447 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:12Z"} 2024-03-04T18:35:12.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:12.447 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:12.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:12.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:35:17.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:17.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:17.449 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:17.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:17.502 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:17.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:17.563+0000 7ff799e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:17.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:17.564+0000 7ff799e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:17.590 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:17Z"} 2024-03-04T18:35:17.590 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:17Z"} 2024-03-04T18:35:17.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:17.591 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:17.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:17.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:35:22.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:22.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:22.593 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:22.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:22.692+0000 7f0992413800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:22.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:22.693+0000 7f0992413800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:22.715 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:22Z"} 2024-03-04T18:35:22.716 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:22Z"} 2024-03-04T18:35:22.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:22.716 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:22.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:22.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.835525_697.1 2024-03-04T18:35:27.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:27.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:27.717 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:27.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:27.765 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:27.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:27.827+0000 7f326cb42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:27.828 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:27.828+0000 7f326cb42800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:27.860 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:27Z"} 2024-03-04T18:35:27.860 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:27Z"} 2024-03-04T18:35:27.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:27.861 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:27.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:27.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T18:35:32.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:32.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:32.864 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:32.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:32.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:32.982 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:32.983+0000 7f288f5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:32.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:32.984+0000 7f288f5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:33.005 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:32Z"} 2024-03-04T18:35:33.005 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:32Z"} 2024-03-04T18:35:33.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:33.005 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:33.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:33.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.218376_725.1 2024-03-04T18:35:38.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:38.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:38.007 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:38.060 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:38.061 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:38.116 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:38.117+0000 7f7c5ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:38.117 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:38.118+0000 7f7c5ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:38.148 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:38Z"} 2024-03-04T18:35:38.149 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:38Z"} 2024-03-04T18:35:38.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:38.149 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:38.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:38.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T18:35:43.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:43.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:43.151 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:43.328 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:43.329+0000 7fc407e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:43.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:43.330+0000 7fc407e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:43.353 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:43Z"} 2024-03-04T18:35:43.353 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:43Z"} 2024-03-04T18:35:43.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:43.354 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:43.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:43.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T18:35:48.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:48.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:48.355 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:48.401 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:48.401 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:48.457 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:48.458+0000 7faaf4c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:48.458 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:48.459+0000 7faaf4c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:48.483 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:48Z"} 2024-03-04T18:35:48.483 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:48Z"} 2024-03-04T18:35:48.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:48.484 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:48.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:48.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.842248_699.1 2024-03-04T18:35:53.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:53.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:53.486 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:53.532 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:53.533 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:53.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:53.591+0000 7f743472b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:53.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:53.592+0000 7f743472b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:53.614 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:53Z"} 2024-03-04T18:35:53.615 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:53Z"} 2024-03-04T18:35:53.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:53.615 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:53.615 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:53.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.692350_570.1 2024-03-04T18:35:58.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:58.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:35:58.617 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:35:58.664 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:35:58.664 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:35:58.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:58.720+0000 7f50d482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:58.720 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:35:58.720+0000 7f50d482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:35:58.753 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:58Z"} 2024-03-04T18:35:58.753 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:58Z"} 2024-03-04T18:35:58.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:35:58.753 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:58.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:35:58.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T18:36:03.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:03.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:03.755 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:03.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:03.808 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:03.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:03.865+0000 7f6c74d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:03.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:03.866+0000 7f6c74d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:03.891 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:03Z"} 2024-03-04T18:36:03.891 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:03Z"} 2024-03-04T18:36:03.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.159615_716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.159615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:03.891 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:03.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.159615_716.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:03.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.159615_716.1 2024-03-04T18:36:08.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:08.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:08.893 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:08.942 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:08.942 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:09.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:09.012+0000 7f7156f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:09.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:09.013+0000 7f7156f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:09.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:09Z"} 2024-03-04T18:36:09.040 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:09Z"} 2024-03-04T18:36:09.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:09.041 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:09.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:09.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:36:14.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:14.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:14.042 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:14.095 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:14.095 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:14.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:14.154+0000 7fead2c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:14.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:14.155+0000 7fead2c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:14.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:14Z"} 2024-03-04T18:36:14.184 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:14Z"} 2024-03-04T18:36:14.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:14.187 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:14.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:14.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:36:19.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:19.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:19.188 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:19.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:19.235 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:19.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:19.302+0000 7f751d014800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:19.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:19.303+0000 7f751d014800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:19.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:19Z"} 2024-03-04T18:36:19.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:19Z"} 2024-03-04T18:36:19.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:19.334 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:19.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:19.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:36:24.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:24.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:24.336 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:24.386 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:24.386 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:24.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:24.445+0000 7effbd62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:24.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:24.446+0000 7effbd62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:24.477 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:24Z"} 2024-03-04T18:36:24.478 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:24Z"} 2024-03-04T18:36:24.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.214996_723.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.214996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:24.478 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:24.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.214996_723.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:24.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.214996_723.1 2024-03-04T18:36:29.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:29.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:29.480 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:29.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:29.531 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:29.703 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:29.703+0000 7fda52e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:29.706 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:29.707+0000 7fda52e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:29Z"} 2024-03-04T18:36:29.759 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:29Z"} 2024-03-04T18:36:29.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:29.760 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:29.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:29.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.996460_704.1 2024-03-04T18:36:34.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:34.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:34.761 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:34.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:34.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:34.861 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:34.862+0000 7f36ab733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:34.862 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:34.863+0000 7f36ab733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:34.895 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:34Z"} 2024-03-04T18:36:34.895 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:34Z"} 2024-03-04T18:36:34.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:34.895 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:34.896 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:34.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T18:36:39.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:39.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:39.897 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:40.017+0000 7f0aee7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:40.018 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:40.018+0000 7f0aee7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:40.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:40Z"} 2024-03-04T18:36:40.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:40Z"} 2024-03-04T18:36:40.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:40.048 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:40.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:40.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:36:45.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:45.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:45.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:45.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:45.097 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:45.159+0000 7fcd9d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:45.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:45.160+0000 7fcd9d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:45.192 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:45Z"} 2024-03-04T18:36:45.192 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:45Z"} 2024-03-04T18:36:45.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:45.193 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:45.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:45.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T18:36:50.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:50.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:50.195 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:50.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:50.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:50.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:50.303+0000 7fac96533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:50.304+0000 7fac96533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:50.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:50Z"} 2024-03-04T18:36:50.330 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:50Z"} 2024-03-04T18:36:50.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:50.330 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:50.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:50.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T18:36:55.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:55.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:36:55.333 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:36:55.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:36:55.381 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:36:55.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:55.429+0000 7f7cc0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:55.430 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:36:55.430+0000 7f7cc0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:36:55.453 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:55Z"} 2024-03-04T18:36:55.453 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:55Z"} 2024-03-04T18:36:55.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:36:55.453 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:55.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:36:55.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T18:37:00.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:00.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:00.455 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:00.501 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:00.501 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:00.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:00.557+0000 7fc2bfa30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:00.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:00.558+0000 7fc2bfa30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:00.581 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:00Z"} 2024-03-04T18:37:00.581 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:00Z"} 2024-03-04T18:37:00.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:00.582 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:00.582 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:00.582 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.154969_713.1 2024-03-04T18:37:05.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:05.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:05.584 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:05.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:05.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:05.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:05.682+0000 7fb993431800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:05.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:05.683+0000 7fb993431800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:05.708 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:05Z"} 2024-03-04T18:37:05.708 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:05Z"} 2024-03-04T18:37:05.708 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:05.708 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:05.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:05.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T18:37:10.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:10.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:10.710 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:10.764 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:10.764 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:10.823 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:10.824+0000 7f1f80c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:10.824 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:10.825+0000 7f1f80c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:10.858 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:10Z"} 2024-03-04T18:37:10.859 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:10Z"} 2024-03-04T18:37:10.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:10.859 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:10.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:10.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:37:15.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:15.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:15.861 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:15.909 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:15.910 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:15.966+0000 7f1ea253b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:15.967+0000 7f1ea253b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:15.991 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:15Z"} 2024-03-04T18:37:15.992 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:15Z"} 2024-03-04T18:37:15.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:15.992 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:15.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:15.992 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T18:37:20.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:20.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:20.994 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:21.048 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:21.048 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:21.109 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:21.110+0000 7f9188339800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:21.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:21.111+0000 7f9188339800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:21.141 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:21Z"} 2024-03-04T18:37:21.142 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:21Z"} 2024-03-04T18:37:21.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:21.142 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:21.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:21.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T18:37:26.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:26.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:26.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:26.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:26.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:26.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:26.263+0000 7f26db94b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:26.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:26.264+0000 7f26db94b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:26.296 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:26Z"} 2024-03-04T18:37:26.296 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:26Z"} 2024-03-04T18:37:26.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:26.297 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:26.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:26.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T18:37:31.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:31.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:31.298 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:31.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:31.349 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:31.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:31.463+0000 7f2f8cf22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:31.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:31.470+0000 7f2f8cf22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:31.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:31Z"} 2024-03-04T18:37:31.550 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:31Z"} 2024-03-04T18:37:31.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:31.550 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:31.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:31.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.842248_699.1 2024-03-04T18:37:36.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:36.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:36.552 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:36.696 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:36.697+0000 7fc822c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:36.697 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:36.697+0000 7fc822c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:36.729 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:36Z"} 2024-03-04T18:37:36.730 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:36Z"} 2024-03-04T18:37:36.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:36.730 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:36.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:36.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.220171_727.1 2024-03-04T18:37:41.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:41.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:41.732 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:41.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:41.778 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:41.835 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:41.835+0000 7f37c868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:41.835 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:41.836+0000 7f37c868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:41.861 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:41Z"} 2024-03-04T18:37:41.861 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:41Z"} 2024-03-04T18:37:41.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:41.862 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:41.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:41.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T18:37:46.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:46.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:46.864 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:46.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:46.913 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:46.978 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:46.978+0000 7f3d92d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:46.979 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:46.979+0000 7f3d92d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:47.002 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:46Z"} 2024-03-04T18:37:47.002 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:46Z"} 2024-03-04T18:37:47.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:47.003 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:47.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:47.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T18:37:52.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:52.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:52.004 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:52.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:52.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:52.112 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:52.112+0000 7f81cf20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:52.112 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:52.113+0000 7f81cf20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:52.139 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:52Z"} 2024-03-04T18:37:52.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:52Z"} 2024-03-04T18:37:52.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:52.140 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:52.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:52.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T18:37:57.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:57.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:37:57.141 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:37:57.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:37:57.189 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:37:57.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:57.239+0000 7fadb0b24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:57.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:37:57.240+0000 7fadb0b24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:37:57.261 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:57Z"} 2024-03-04T18:37:57.262 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:57Z"} 2024-03-04T18:37:57.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:37:57.262 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:57.262 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:37:57.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T18:38:02.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:02.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:02.264 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:02.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:02.313 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:02.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:02.374+0000 7fcb0a90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:02.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:02.375+0000 7fcb0a90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:02.404 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:02Z"} 2024-03-04T18:38:02.405 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:02Z"} 2024-03-04T18:38:02.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:02.405 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:02.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:02.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:38:07.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:07.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:07.407 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:07.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:07.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:07.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:07.531+0000 7f5d6dd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:07.531 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:07.532+0000 7f5d6dd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:07.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:07Z"} 2024-03-04T18:38:07.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:07Z"} 2024-03-04T18:38:07.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:07.561 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:07.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:07.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.987908_703.1 2024-03-04T18:38:12.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:12.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:12.563 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:12.615 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:12.615 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:12.677+0000 7fbd18133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:12.678+0000 7fbd18133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:12.712 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:12Z"} 2024-03-04T18:38:12.713 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:12Z"} 2024-03-04T18:38:12.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:12.713 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:12.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:12.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T18:38:17.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:17.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:17.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:17.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:17.768 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:17.824 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:17.825+0000 7f04aa48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:17.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:17.826+0000 7f04aa48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:17.848 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:17Z"} 2024-03-04T18:38:17.848 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:17Z"} 2024-03-04T18:38:17.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:17.849 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:17.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:17.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T18:38:22.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:22.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:22.851 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:22.904 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:22.904 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:22.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:22.961+0000 7f6dd5e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:22.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:22.962+0000 7f6dd5e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:23.025 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:22Z"} 2024-03-04T18:38:23.025 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:22Z"} 2024-03-04T18:38:23.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:23.026 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:23.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:23.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.996460_704.1 2024-03-04T18:38:28.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:28.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:28.027 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:28.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:28.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:28.179 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:28.179+0000 7fe30120f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:28.179 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:28.180+0000 7fe30120f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:28.203 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:28Z"} 2024-03-04T18:38:28.203 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:28Z"} 2024-03-04T18:38:28.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:28.203 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:28.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:28.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T18:38:33.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:33.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:33.205 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:33.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:33.276 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:33.335 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:33.336+0000 7fce1e613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:33.336 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:33.337+0000 7fce1e613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:33.359 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:33Z"} 2024-03-04T18:38:33.360 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:33Z"} 2024-03-04T18:38:33.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:33.360 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:33.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:33.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T18:38:38.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:38.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:38.362 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:38.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:38.408 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:38.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:38.469+0000 7efe5c90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:38.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:38.470+0000 7efe5c90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:38.495 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:38Z"} 2024-03-04T18:38:38.495 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:38Z"} 2024-03-04T18:38:38.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:38.496 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:38.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:38.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T18:38:43.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:43.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:43.497 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:43.545 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:43.545 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:43.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:43.598+0000 7f497bf1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:43.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:43.599+0000 7f497bf1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:43.620 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:43Z"} 2024-03-04T18:38:43.621 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:43Z"} 2024-03-04T18:38:43.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:43.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:43.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:43.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.987908_703.1 2024-03-04T18:38:48.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:48.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:48.622 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:48.668 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:48.669 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:48.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:48.724+0000 7f1fe1526800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:48.724 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:48.725+0000 7f1fe1526800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:48.748 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:48Z"} 2024-03-04T18:38:48.748 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:48Z"} 2024-03-04T18:38:48.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:48.748 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:48.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:48.749 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:38:53.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:53.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:53.750 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:53.798 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:53.798 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:53.859 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:53.859+0000 7fa40842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:53.860 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:53.860+0000 7fa40842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:53.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:53Z"} 2024-03-04T18:38:53.884 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:53Z"} 2024-03-04T18:38:53.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:53.885 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:53.885 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:53.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:38:58.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:58.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:38:58.887 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:38:58.934 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:38:58.934 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:38:58.992 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:58.993+0000 7f0ae0a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:58.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:38:58.994+0000 7f0ae0a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:38:59.023 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:58Z"} 2024-03-04T18:38:59.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:58Z"} 2024-03-04T18:38:59.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:38:59.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:59.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:38:59.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.121915_709.1 2024-03-04T18:39:04.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:04.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:04.025 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:04.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:04.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:04.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:04.123+0000 7f405ac1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:04.123 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:04.124+0000 7f405ac1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:04.152 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:04Z"} 2024-03-04T18:39:04.152 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:04Z"} 2024-03-04T18:39:04.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:04.153 DEBUG:rgw_multi.tests:log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:04.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:04.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T18:39:09.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-03-04T18:39:09.157 DEBUG:boto:path=/sbnvju-78/ 2024-03-04T18:39:09.157 DEBUG:boto:auth_path=/sbnvju-78/ 2024-03-04T18:39:09.157 DEBUG:boto:Method: PUT 2024-03-04T18:39:09.157 DEBUG:boto:Path: /sbnvju-78/ 2024-03-04T18:39:09.157 DEBUG:boto:Data: 2024-03-04T18:39:09.157 DEBUG:boto:Headers: {} 2024-03-04T18:39:09.158 DEBUG:boto:Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T18:39:09.158 DEBUG:boto:Port: 8000 2024-03-04T18:39:09.158 DEBUG:boto:Params: {} 2024-03-04T18:39:09.158 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T18:39:09.158 DEBUG:boto:Token: None 2024-03-04T18:39:09.159 DEBUG:boto:StringToSign: PUT Mon, 04 Mar 2024 18:39:09 GMT /sbnvju-78/ 2024-03-04T18:39:09.159 DEBUG:boto:Signature: AWS 4LUC3XJ67HIPQ3U1NKIY:d5itYgP7qYLTprx7D/c03hzsgpA= 2024-03-04T18:39:09.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:39:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d5itYgP7qYLTprx7D/c03hzsgpA=', 'Content-Length': '0'} 2024-03-04T18:39:09.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007fa02eae5c329518-0065e6154d-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T18:39:09.183 DEBUG:rgw_multi.tests:created bucket=sbnvju-78 2024-03-04T18:39:09.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:09.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:09.183 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:39:09.312 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:39:09.312 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:09.312 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:39:09.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:09.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:09.313 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:09.499 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:09.500+0000 7ff97772f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:09.500 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:09.500+0000 7ff97772f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:09.532 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:09Z"} 2024-03-04T18:39:09.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:09Z"} 2024-03-04T18:39:09.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:09.533 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:09.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:09.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:39:14.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:14.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:14.535 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:14.581 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:14.581 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:14.641 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:14.642+0000 7fc36d330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:14.642 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:14.642+0000 7fc36d330800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:14Z"} 2024-03-04T18:39:14.666 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:14Z"} 2024-03-04T18:39:14.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:14.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:14.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:14.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:39:19.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:19.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:19.669 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:19.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:19.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:19.863 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:19.863+0000 7f1d4793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:19.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:19.866+0000 7f1d4793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:19.929 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:19Z"} 2024-03-04T18:39:19.929 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:19Z"} 2024-03-04T18:39:19.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:19.930 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:19.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:19.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:39:24.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:24.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:24.931 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:24.978 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:24.978 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:25.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:25.033+0000 7f5cd829d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:25.034 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:25.034+0000 7f5cd829d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:25.061 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:25Z"} 2024-03-04T18:39:25.062 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:25Z"} 2024-03-04T18:39:25.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:25.063 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:25.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:25.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:39:30.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:30.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:30.064 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:30.110 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:30.111 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:30.175 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:30.176+0000 7f4387430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:30.176 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:30.177+0000 7f4387430800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:30.201 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:30Z"} 2024-03-04T18:39:30.201 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:30Z"} 2024-03-04T18:39:30.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:30.201 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:30.201 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:30.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:39:35.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:35.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:35.203 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:35.300+0000 7fbe9052d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:35.300 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:35.301+0000 7fbe9052d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:35.324 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:35Z"} 2024-03-04T18:39:35.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:35Z"} 2024-03-04T18:39:35.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:35.324 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:35.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:35.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:39:40.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:40.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:40.326 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:40.372 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:40.373 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:40.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:40.435+0000 7f6cc692f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:40.435 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:40.436+0000 7f6cc692f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:40.460 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:40Z"} 2024-03-04T18:39:40.460 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:40Z"} 2024-03-04T18:39:40.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:40.460 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:40.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:40.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:39:45.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:45.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:45.462 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:45.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:45.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:45.557+0000 7f9a61f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:45.558+0000 7f9a61f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:45.579 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:45Z"} 2024-03-04T18:39:45.580 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:45Z"} 2024-03-04T18:39:45.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:45.580 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:45.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:45.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:39:50.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:50.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:50.582 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:50.630 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:50.630 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:50.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:50.690+0000 7fc9b31a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:50.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:50.702+0000 7fc9b31a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:50.798 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:50Z"} 2024-03-04T18:39:50.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:50Z"} 2024-03-04T18:39:50.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:50.799 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:50.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:50.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.175444_719.1 2024-03-04T18:39:55.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:55.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:39:55.801 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:39:55.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:39:55.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:39:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:55.931+0000 7f8a05d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:55.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:39:55.932+0000 7f8a05d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:39:55.954 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:55Z"} 2024-03-04T18:39:55.954 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:55Z"} 2024-03-04T18:39:55.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:39:55.955 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:55.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:39:55.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:40:00.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:00.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:00.957 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:01.003 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:01.004 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:01.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:01.056+0000 7f0c6d8c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:01.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:01.057+0000 7f0c6d8c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:01.083 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:01Z"} 2024-03-04T18:40:01.083 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:01Z"} 2024-03-04T18:40:01.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:01.084 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:01.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:01.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:40:06.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:06.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:06.085 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:06.131 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:06.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:06.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:06.193+0000 7f891a92a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:06.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:06.193+0000 7f891a92a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:06.217 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:06Z"} 2024-03-04T18:40:06.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:06Z"} 2024-03-04T18:40:06.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:06.218 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:06.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:06.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:40:11.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:11.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:11.219 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:11.267 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:11.267 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:11.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:11.315+0000 7f1b0608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:11.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:11.316+0000 7f1b0608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:11.338 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:11Z"} 2024-03-04T18:40:11.338 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:11Z"} 2024-03-04T18:40:11.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:11.339 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:11.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:11.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:40:16.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:16.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:16.340 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:16.389 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:16.389 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:16.502 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:16.503+0000 7f2411e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:16.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:16.506+0000 7f2411e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:16.570 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:16Z"} 2024-03-04T18:40:16.570 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:16Z"} 2024-03-04T18:40:16.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:16.570 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:16.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:16.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T18:40:21.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:21.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:21.572 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:21.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:21.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:21.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:21.781+0000 7f90bdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:21.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:21.787+0000 7f90bdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:21.842 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:21Z"} 2024-03-04T18:40:21.842 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:21Z"} 2024-03-04T18:40:21.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:21.842 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:21.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:21.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T18:40:26.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:26.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:26.844 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:26.903 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:26.903 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:26.963+0000 7f3b4c613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:26.964+0000 7f3b4c613800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:26.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:26Z"} 2024-03-04T18:40:26.986 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:26Z"} 2024-03-04T18:40:26.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:26.987 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:26.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:26.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:40:31.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:31.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:31.988 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:32.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:32.034 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:32.109 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:32.109+0000 7fb917911800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:32.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:32.110+0000 7fb917911800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:32.143 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.245429_730.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.245429Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:32Z"} 2024-03-04T18:40:32.143 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.245429_730.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.245429Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:32Z"} 2024-03-04T18:40:32.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.245429_730.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.245429Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:32.144 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:32.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.245429_730.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:32.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.245429_730.1 2024-03-04T18:40:37.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:37.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:37.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:37.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:37.192 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:37.251 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:37.252+0000 7fbb29e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:37.252 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:37.253+0000 7fbb29e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:37.286 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:37Z"} 2024-03-04T18:40:37.286 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:37Z"} 2024-03-04T18:40:37.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:37.287 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:37.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:37.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:40:42.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:42.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:42.288 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:42.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:42.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:42.400 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:42.401+0000 7f19c6a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:42.401 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:42.402+0000 7f19c6a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:42.429 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:42Z"} 2024-03-04T18:40:42.429 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:42Z"} 2024-03-04T18:40:42.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:42.429 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:42.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:42.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T18:40:47.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:47.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:47.432 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:47.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:47.478 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:47.532 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:47.532+0000 7f9ced41b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:47.532 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:47.533+0000 7f9ced41b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:47.555 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:47Z"} 2024-03-04T18:40:47.555 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:47Z"} 2024-03-04T18:40:47.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:47.555 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:47.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:47.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:40:52.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:52.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:52.558 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:52.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:52.605 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:52.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:52.660+0000 7f781d612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:52.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:52.660+0000 7f781d612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:52.683 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:52Z"} 2024-03-04T18:40:52.683 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:52Z"} 2024-03-04T18:40:52.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:52.683 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:52.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:52.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:40:57.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:57.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:40:57.685 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:40:57.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:40:57.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:40:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:57.788+0000 7f951093c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:57.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:40:57.788+0000 7f951093c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:40:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:57Z"} 2024-03-04T18:40:57.819 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:57Z"} 2024-03-04T18:40:57.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:40:57.819 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:57.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:40:57.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:41:02.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:02.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:02.821 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:02.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:02.869 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:02.937 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:02.937+0000 7fbaa3ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:02.937 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:02.938+0000 7fbaa3ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:02.964 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:02Z"} 2024-03-04T18:41:02.965 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:02Z"} 2024-03-04T18:41:02.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:02.965 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:02.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:02.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:41:07.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:07.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:07.967 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:08.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:08.021 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:08.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:08.092+0000 7f6a7f815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:08.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:08.093+0000 7f6a7f815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:08.126 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:08Z"} 2024-03-04T18:41:08.126 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:08Z"} 2024-03-04T18:41:08.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:08.126 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:08.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:08.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:41:13.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:13.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:13.128 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:13.235 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:13.234+0000 7f6b8cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:13.235 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:13.235+0000 7f6b8cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:13.256 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:13Z"} 2024-03-04T18:41:13.257 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:13Z"} 2024-03-04T18:41:13.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:13.257 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:13.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:13.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:41:18.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:18.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:18.258 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:18.311 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:18.312 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:18.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:18.373+0000 7f8b79a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:18.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:18.374+0000 7f8b79a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:18.398 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:18Z"} 2024-03-04T18:41:18.398 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:18Z"} 2024-03-04T18:41:18.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:18.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:18.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:18.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:41:23.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:23.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:23.402 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:23.454 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:23.454 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:23.507 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:23.508+0000 7f5b43542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:23.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:23.509+0000 7f5b43542800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:23.532 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:23Z"} 2024-03-04T18:41:23.533 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:23Z"} 2024-03-04T18:41:23.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:23.533 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:23.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:23.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:41:28.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:28.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:28.535 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:28.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:28.662+0000 7f419cbaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:28.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:28.663+0000 7f419cbaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:28.688 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:28Z"} 2024-03-04T18:41:28.688 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:28Z"} 2024-03-04T18:41:28.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:28.689 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:28.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:28.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:41:33.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:33.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:33.690 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:33.743 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:33.743 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:33.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:33.797+0000 7fcc14333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:33.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:33.798+0000 7fcc14333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:33.823 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:33Z"} 2024-03-04T18:41:33.823 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:33Z"} 2024-03-04T18:41:33.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:33.823 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:33.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:33.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:41:38.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:38.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:38.825 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:38.878 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:38.879 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:39.032 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:39.033+0000 7fc652b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:39.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:39.034+0000 7fc652b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:39.083 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:39Z"} 2024-03-04T18:41:39.083 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:39Z"} 2024-03-04T18:41:39.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:39.084 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:39.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:39.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:41:44.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:44.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:44.086 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:44.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:44.133 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:44.181 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:44.181+0000 7fd76849d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:44.181 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:44.182+0000 7fd76849d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:44.206 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:44Z"} 2024-03-04T18:41:44.206 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:44Z"} 2024-03-04T18:41:44.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:44.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:44.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:44.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:41:49.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:49.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:49.208 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:49.254 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:49.255 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:49.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:49.317+0000 7ff00aa24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:49.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:49.318+0000 7ff00aa24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:49.338 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:49Z"} 2024-03-04T18:41:49.339 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:49Z"} 2024-03-04T18:41:49.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:49.339 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:49.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:49.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:41:54.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:54.341 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:54.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:54.388 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:54.458+0000 7f0ecc436800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:54.459+0000 7f0ecc436800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:54.487 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:54Z"} 2024-03-04T18:41:54.488 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:54Z"} 2024-03-04T18:41:54.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:54.488 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:54.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:54.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:41:59.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:59.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:41:59.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:41:59.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:41:59.546 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:41:59.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:59.598+0000 7f7d9b3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:59.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:41:59.599+0000 7f7d9b3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:41:59.625 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:59Z"} 2024-03-04T18:41:59.625 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:59Z"} 2024-03-04T18:41:59.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:41:59.626 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:59.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:41:59.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:42:04.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:04.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:04.628 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:04.676 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:04.676 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:04.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:04.744+0000 7f2d93a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:04.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:04.745+0000 7f2d93a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:04.772 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:04Z"} 2024-03-04T18:42:04.773 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:04Z"} 2024-03-04T18:42:04.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:04.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:04.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:04.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:42:09.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:09.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:09.775 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:09.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:09.822 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:09.892 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:09.893+0000 7ff118f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:09.893 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:09.894+0000 7ff118f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:09.920 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:09Z"} 2024-03-04T18:42:09.920 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:09Z"} 2024-03-04T18:42:09.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:09.921 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:09.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:09.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:42:14.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:14.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:14.922 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:14.972 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:14.972 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:15.043 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:15.044+0000 7f65adc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:15.044 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:15.045+0000 7f65adc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:15.075 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:15Z"} 2024-03-04T18:42:15.075 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:15Z"} 2024-03-04T18:42:15.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:15.076 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:15.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:15.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:42:20.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:20.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:20.077 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:20.125 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:20.125 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:20.175 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:20.175+0000 7fd1f5d2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:20.175 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:20.176+0000 7fd1f5d2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:20.198 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:20Z"} 2024-03-04T18:42:20.198 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:20Z"} 2024-03-04T18:42:20.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:20.198 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:20.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:20.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:42:25.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:25.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:25.200 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:25.254 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:25.254 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:25.313+0000 7f82bbd2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:25.314+0000 7f82bbd2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:25.343 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:25Z"} 2024-03-04T18:42:25.343 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:25Z"} 2024-03-04T18:42:25.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:25.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:25.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:25.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:42:30.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:30.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:30.345 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:30.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:30.465+0000 7fd4fb73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:30.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:30.466+0000 7fd4fb73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:30.554 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:30Z"} 2024-03-04T18:42:30.554 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:30Z"} 2024-03-04T18:42:30.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:30.554 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:30.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:30.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:42:35.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:35.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:35.556 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:35.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:35.602 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:35.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:35.655+0000 7f5cc3c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:35.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:35.656+0000 7f5cc3c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:35.681 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:35Z"} 2024-03-04T18:42:35.681 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:35Z"} 2024-03-04T18:42:35.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:35.681 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:35.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:35.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:42:40.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:40.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:40.683 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:40.786+0000 7fba5bb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:40.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:40.787+0000 7fba5bb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:40.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:40Z"} 2024-03-04T18:42:40.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:40Z"} 2024-03-04T18:42:40.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:40.809 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:40.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:40.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:42:45.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:45.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:45.811 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:45.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:45.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:45.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:45.911+0000 7f352a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:45.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:45.912+0000 7f352a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:45.933 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:45Z"} 2024-03-04T18:42:45.933 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:45Z"} 2024-03-04T18:42:45.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:45.933 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:45.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:45.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T18:42:50.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:50.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:50.936 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:50.984 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:50.984 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:51.042+0000 7fa5f6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:51.042 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:51.043+0000 7fa5f6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:51.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:51Z"} 2024-03-04T18:42:51.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:51Z"} 2024-03-04T18:42:51.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:51.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:51.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:51.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:42:56.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:56.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:42:56.069 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:42:56.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:42:56.123 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:42:56.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:56.186+0000 7f1188a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:56.186 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:42:56.187+0000 7f1188a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:42:56.214 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:56Z"} 2024-03-04T18:42:56.215 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:56Z"} 2024-03-04T18:42:56.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:42:56.215 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:56.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:42:56.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:43:01.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:01.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:01.217 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:01.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:01.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:01.325 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:01.326+0000 7f1782e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:01.326 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:01.327+0000 7f1782e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:01.350 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:01Z"} 2024-03-04T18:43:01.350 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:01Z"} 2024-03-04T18:43:01.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:01.350 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:01.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:01.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:43:06.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:06.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:06.352 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:06.401 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:06.401 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:06.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:06.465+0000 7f541931e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:06.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:06.466+0000 7f541931e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:06.491 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:06Z"} 2024-03-04T18:43:06.491 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:06Z"} 2024-03-04T18:43:06.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:06.491 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:06.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:06.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:43:11.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:11.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:11.493 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:11.544 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:11.544 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:11.598 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:11.598+0000 7f3860b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:11.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:11.599+0000 7f3860b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:11.622 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:11Z"} 2024-03-04T18:43:11.622 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:11Z"} 2024-03-04T18:43:11.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:11.622 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:11.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:11.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T18:43:16.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:16.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:16.624 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:16.676 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:16.676 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:16.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:16.732+0000 7f138713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:16.732 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:16.733+0000 7f138713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:16.755 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:16Z"} 2024-03-04T18:43:16.756 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:16Z"} 2024-03-04T18:43:16.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:16.756 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:16.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:16.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:43:21.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:21.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:21.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:21.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:21.807 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:21.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:21.879+0000 7f51b4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:21.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:21.879+0000 7f51b4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:21.903 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:21Z"} 2024-03-04T18:43:21.903 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:21Z"} 2024-03-04T18:43:21.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:21.904 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:21.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:21.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:43:26.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:26.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:26.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:26.960 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:26.960 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:27.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:27.036+0000 7f530d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:27.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:27.037+0000 7f530d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:27.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:27Z"} 2024-03-04T18:43:27.066 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:27Z"} 2024-03-04T18:43:27.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:27.066 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:27.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:27.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:43:32.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:32.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:32.067 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:32.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:32.121 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:32.178 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:32.179+0000 7f855352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:32.179 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:32.180+0000 7f855352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:32.202 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:32Z"} 2024-03-04T18:43:32.202 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:32Z"} 2024-03-04T18:43:32.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:32.203 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:32.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:32.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:43:37.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:37.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:37.205 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:37.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:37.258 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:37.318 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:37.319+0000 7f489d731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:37.319 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:37.320+0000 7f489d731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:37.343 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:37Z"} 2024-03-04T18:43:37.344 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:37Z"} 2024-03-04T18:43:37.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:37.344 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:37.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:37.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:43:42.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:42.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:42.346 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:42.399 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:42.399 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:42.546 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:42.548+0000 7fcaf869d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:42.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:42.551+0000 7fcaf869d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:42.617 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:42Z"} 2024-03-04T18:43:42.617 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:42Z"} 2024-03-04T18:43:42.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:42.617 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:42.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:42.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:43:47.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:47.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:47.619 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:47.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:47.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:47.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:47.737+0000 7f42f1d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:47.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:47.738+0000 7f42f1d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:47.771 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:47Z"} 2024-03-04T18:43:47.772 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:47Z"} 2024-03-04T18:43:47.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:47.772 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:47.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:47.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:43:52.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:52.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:52.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:52.869 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:52.869+0000 7f46a008c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:52.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:52.870+0000 7f46a008c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:52.893 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:52Z"} 2024-03-04T18:43:52.894 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:52Z"} 2024-03-04T18:43:52.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:52.894 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:52.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:52.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:43:57.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:57.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:43:57.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:43:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:43:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:43:58.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:58.005+0000 7f376734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:58.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:43:58.006+0000 7f376734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:43:58.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:58Z"} 2024-03-04T18:43:58.034 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:58Z"} 2024-03-04T18:43:58.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:43:58.034 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:58.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:43:58.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T18:44:03.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:03.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:03.036 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:03.132+0000 7f8caa51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:03.133+0000 7f8caa51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:03.160 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:03Z"} 2024-03-04T18:44:03.161 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:03Z"} 2024-03-04T18:44:03.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:03.161 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:03.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:03.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:44:08.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:08.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:08.162 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:08.215 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:08.216 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:08.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:08.274+0000 7f463ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:08.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:08.276+0000 7f463ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:08.299 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:08Z"} 2024-03-04T18:44:08.300 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:08Z"} 2024-03-04T18:44:08.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:08.300 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:08.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:08.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:44:13.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:13.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:13.302 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:13.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:13.350 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:13.401 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:13.402+0000 7f2dbf40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:13.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:13.403+0000 7f2dbf40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:13.424 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:13Z"} 2024-03-04T18:44:13.425 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:13Z"} 2024-03-04T18:44:13.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:13.425 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:13.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:13.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:44:18.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-03-04T18:44:18.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:18.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:18.430 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:18.483 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:18.483 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:18.534 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:44:18.556 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:44:18.556 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:18.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:44:18.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:18.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:18.557 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:18.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:18.610 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:18.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:18.675+0000 7f3bac017800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:18.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:18.676+0000 7f3bac017800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:18.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:18Z"} 2024-03-04T18:44:18.702 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:18Z"} 2024-03-04T18:44:18.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:18.703 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:18.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:18.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:44:23.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:23.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:23.704 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:23.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:23.752 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:23.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:23.810+0000 7f9ee2a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:23.811+0000 7f9ee2a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:23.833 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:23Z"} 2024-03-04T18:44:23.833 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:23Z"} 2024-03-04T18:44:23.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:23.833 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:23.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:23.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:44:28.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:28.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:28.835 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:28.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:28.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:28.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:28.940+0000 7fadd3731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:28.940 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:28.941+0000 7fadd3731800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:28.964 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:28Z"} 2024-03-04T18:44:28.964 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:28Z"} 2024-03-04T18:44:28.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:28.965 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:28.965 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:28.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:44:33.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:33.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:33.966 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:34.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:34.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:34.080 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:34.080+0000 7fd1f1024800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:34.081 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:34.081+0000 7fd1f1024800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:34.108 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:34Z"} 2024-03-04T18:44:34.108 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:34Z"} 2024-03-04T18:44:34.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:34.109 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:34.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:34.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:44:39.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:39.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:39.110 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:39.190 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:39.190 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:39.301 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:39.301+0000 7f8d12d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:39.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:39.302+0000 7f8d12d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:39.326 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:39Z"} 2024-03-04T18:44:39.326 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:39Z"} 2024-03-04T18:44:39.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:39.327 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:39.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:39.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T18:44:44.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:44.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:44.329 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:44.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:44.375 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:44.436 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:44.436+0000 7feed3017800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:44.436 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:44.437+0000 7feed3017800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:44.461 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:44Z"} 2024-03-04T18:44:44.461 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:44Z"} 2024-03-04T18:44:44.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:44.461 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:44.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:44.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T18:44:49.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:49.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:49.463 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:49.508 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:49.509 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:49.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:49.574+0000 7f8cdbe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:49.574 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:49.575+0000 7f8cdbe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:49.599 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:49Z"} 2024-03-04T18:44:49.599 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:49Z"} 2024-03-04T18:44:49.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:49.600 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:49.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:49.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:44:54.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:54.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:54.601 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:54.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:54.647 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:54.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:54.698+0000 7fbb677a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:54.698 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:54.699+0000 7fbb677a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:54.723 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:54Z"} 2024-03-04T18:44:54.723 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:54Z"} 2024-03-04T18:44:54.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:54.724 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:54.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:54.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:44:59.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:59.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:44:59.726 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:44:59.773 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:44:59.773 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:44:59.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:59.849+0000 7f807e40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:59.849 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:44:59.850+0000 7f807e40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:44:59.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:59Z"} 2024-03-04T18:44:59.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:59Z"} 2024-03-04T18:44:59.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:44:59.878 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:59.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:44:59.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:45:04.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:04.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:04.880 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:04.927 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:04.928 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:04.978 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:04.978+0000 7fb2f8f26800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:04.978 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:04.979+0000 7fb2f8f26800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:05.001 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:04Z"} 2024-03-04T18:45:05.001 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:04Z"} 2024-03-04T18:45:05.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:05.001 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:05.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:05.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.154969_713.1 2024-03-04T18:45:10.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:10.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:10.004 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:10.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:10.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:10.117 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:10.117+0000 7fb92e82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:10.118 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:10.118+0000 7fb92e82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:10.150 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:10Z"} 2024-03-04T18:45:10.150 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:10Z"} 2024-03-04T18:45:10.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:10.150 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:10.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:10.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:45:15.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:15.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:15.152 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:15.201 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:15.201 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:15.261+0000 7f4169f22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:15.262+0000 7f4169f22800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:15.284 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:15Z"} 2024-03-04T18:45:15.284 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:15Z"} 2024-03-04T18:45:15.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:15.285 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:15.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:15.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:45:20.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:20.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:20.287 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:20.335 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:20.335 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:20.386 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:20.386+0000 7f9a36016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:20.386 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:20.387+0000 7f9a36016800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:20.412 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:20Z"} 2024-03-04T18:45:20.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:20Z"} 2024-03-04T18:45:20.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:20.413 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:20.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:20.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:45:25.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:25.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:25.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:25.463 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:25.463 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:25.535 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:25.536+0000 7f9dfeb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:25.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:25.537+0000 7f9dfeb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:25.561 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:25Z"} 2024-03-04T18:45:25.561 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:25Z"} 2024-03-04T18:45:25.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:25.562 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:25.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:25.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:45:30.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:30.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:30.564 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:30.617 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:30.618 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:30.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:30.687+0000 7fef7b92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:30.686 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:30.688+0000 7fef7b92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:30.713 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:30Z"} 2024-03-04T18:45:30.714 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:30Z"} 2024-03-04T18:45:30.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:30.714 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:30.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:30.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:45:35.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:35.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:35.716 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:35.767 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:35.767 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:35.841+0000 7f852df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:35.842+0000 7f852df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:35.871 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:35Z"} 2024-03-04T18:45:35.872 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:35Z"} 2024-03-04T18:45:35.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:35.872 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:35.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:35.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:45:40.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:40.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:40.874 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:40.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:40.921 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:40.988 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:40.988+0000 7fdd907a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:40.989 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:40.989+0000 7fdd907a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:41.012 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:40Z"} 2024-03-04T18:45:41.012 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:40Z"} 2024-03-04T18:45:41.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:41.013 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:41.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:41.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:45:46.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:46.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:46.014 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:46.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:46.093 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:46.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:46.201+0000 7f50aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:46.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:46.202+0000 7f50aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:46.227 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:46Z"} 2024-03-04T18:45:46.227 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:46Z"} 2024-03-04T18:45:46.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:46.227 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:46.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:46.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:45:51.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:51.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:51.229 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:51.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:51.297 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:51.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:51.362+0000 7f2d7de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:51.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:51.362+0000 7f2d7de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:51.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:51Z"} 2024-03-04T18:45:51.386 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:51Z"} 2024-03-04T18:45:51.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:51.386 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:51.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:51.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:45:56.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:56.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:45:56.389 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:45:56.435 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:45:56.435 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:45:56.507 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:56.508+0000 7f72e61af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:56.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:45:56.508+0000 7f72e61af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:45:56.531 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:56Z"} 2024-03-04T18:45:56.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:56Z"} 2024-03-04T18:45:56.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:45:56.532 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:56.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:45:56.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:46:01.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:01.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:01.533 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:01.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:01.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:01.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:01.645+0000 7fc2c7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:01.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:01.646+0000 7fc2c7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:01.674 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:01Z"} 2024-03-04T18:46:01.674 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:01Z"} 2024-03-04T18:46:01.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:01.675 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:01.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:01.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:46:06.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:06.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:06.676 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:06.721 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:06.722 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:06.771 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:06.772+0000 7ff29292d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:06.773+0000 7ff29292d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:06.804 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:06Z"} 2024-03-04T18:46:06.804 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:06Z"} 2024-03-04T18:46:06.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:06.804 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:06.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:06.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T18:46:11.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:11.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:11.806 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:11.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:11.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:11.915 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:11.915+0000 7f8bedd39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:11.916 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:11.916+0000 7f8bedd39800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:11.948 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:11Z"} 2024-03-04T18:46:11.948 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:11Z"} 2024-03-04T18:46:11.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:11.949 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:11.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:11.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:46:16.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:16.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:16.951 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:16.999 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:16.999 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:17.062 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:17.063+0000 7f5e9731e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:17.063 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:17.064+0000 7f5e9731e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:17.095 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.714481_766.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.714481Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:17Z"} 2024-03-04T18:46:17.095 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.714481_766.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.714481Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:17Z"} 2024-03-04T18:46:17.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572290.714481_766.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:11:30.714481Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:17.095 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:17.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1709572290.714481_766.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:17.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709572290.714481_766.1 2024-03-04T18:46:22.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:22.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:22.097 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:22.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:22.143 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:22.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:22.193+0000 7f7397244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:22.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:22.193+0000 7f7397244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:22.217 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:22Z"} 2024-03-04T18:46:22.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:22Z"} 2024-03-04T18:46:22.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:22.217 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:22.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:22.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:46:27.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:27.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:27.219 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:27.265 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:27.266 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:27.324 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:27.326+0000 7f297ca12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:27.325 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:27.327+0000 7f297ca12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:27.349 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:27Z"} 2024-03-04T18:46:27.349 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:27Z"} 2024-03-04T18:46:27.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:27.349 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:27.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:27.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:46:32.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:32.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:32.353 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:32.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:32.400 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:32.452 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:32.453+0000 7f0ab89a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:32.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:32.453+0000 7f0ab89a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:32.477 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.383476_745.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.383476Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:32Z"} 2024-03-04T18:46:32.477 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.383476_745.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.383476Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:32Z"} 2024-03-04T18:46:32.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.383476_745.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.383476Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:32.477 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:32.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.383476_745.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:32.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.383476_745.1 2024-03-04T18:46:37.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:37.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:37.480 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:37.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:37.592+0000 7f8e0422f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:37.592 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:37.593+0000 7f8e0422f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:37.618 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:37Z"} 2024-03-04T18:46:37.618 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:37Z"} 2024-03-04T18:46:37.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:37.619 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:37.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:37.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T18:46:42.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:42.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:42.620 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:42.673 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:42.673 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:42.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:42.736+0000 7f7570239800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:42.737 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:42.737+0000 7f7570239800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:42.772 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:42Z"} 2024-03-04T18:46:42.772 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:42Z"} 2024-03-04T18:46:42.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:42.773 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:42.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:42.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:46:47.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:47.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:47.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:47.903 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:47.903+0000 7f7e9571f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:47.903 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:47.904+0000 7f7e9571f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:47.928 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:47Z"} 2024-03-04T18:46:47.928 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:47Z"} 2024-03-04T18:46:47.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:47.929 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:47.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:47.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:46:52.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:52.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:52.931 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:52.983 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:52.983 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:53.043 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:53.043+0000 7fa109b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:53.044 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:53.044+0000 7fa109b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:53.069 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:53Z"} 2024-03-04T18:46:53.070 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:53Z"} 2024-03-04T18:46:53.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:53.070 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:53.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:53.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:46:58.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:58.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:46:58.072 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:46:58.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:46:58.122 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:46:58.249 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:58.249+0000 7f005d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:58.251 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:46:58.251+0000 7f005d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:46:58.351 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:58Z"} 2024-03-04T18:46:58.352 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:58Z"} 2024-03-04T18:46:58.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:46:58.352 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:58.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:46:58.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:47:03.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:03.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:03.354 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:03.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:03.434 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:03.533 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:03.533+0000 7f512793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:03.534+0000 7f512793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:03.558 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:03Z"} 2024-03-04T18:47:03.558 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:03Z"} 2024-03-04T18:47:03.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:03.559 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:03.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:03.559 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.002120_706.1 2024-03-04T18:47:08.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:08.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:08.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:08.624 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:08.624 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:08.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:08.683+0000 7fc48cd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:08.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:08.684+0000 7fc48cd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:08.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:08Z"} 2024-03-04T18:47:08.706 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:08Z"} 2024-03-04T18:47:08.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:08.707 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:08.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:08.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:47:13.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:13.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:13.708 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:13.754 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:13.754 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:13.803+0000 7f42f8e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:13.804+0000 7f42f8e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:13Z"} 2024-03-04T18:47:13.826 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:13Z"} 2024-03-04T18:47:13.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:13.826 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:13.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:13.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:47:18.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:18.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:18.828 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:18.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:18.935+0000 7f3fdd529800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:18.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:18.936+0000 7f3fdd529800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:18.962 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:18Z"} 2024-03-04T18:47:18.962 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:18Z"} 2024-03-04T18:47:18.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:18.963 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:18.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:18.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:47:23.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:23.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:23.965 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:24.013 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:24.013 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:24.077 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:24.079+0000 7fcf7b7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:24.080+0000 7fcf7b7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:24.112 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:24Z"} 2024-03-04T18:47:24.112 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:24Z"} 2024-03-04T18:47:24.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:24.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:24.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:24.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:47:29.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:29.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:29.115 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:29.163 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:29.163 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:29.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:29.212+0000 7fc82b212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:29.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:29.213+0000 7fc82b212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:29.235 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:29Z"} 2024-03-04T18:47:29.235 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:29Z"} 2024-03-04T18:47:29.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:29.235 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:29.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:29.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T18:47:34.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:34.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:34.238 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:34.342 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:34.343+0000 7f73a3a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:34.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:34.344+0000 7f73a3a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:34.368 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:34Z"} 2024-03-04T18:47:34.368 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:34Z"} 2024-03-04T18:47:34.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:34.369 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:34.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:34.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:47:39.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:39.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:39.370 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:39.417 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:39.417 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:39.476 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:39.476+0000 7f57c0c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:39.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:39.477+0000 7f57c0c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:39.501 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:39Z"} 2024-03-04T18:47:39.501 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:39Z"} 2024-03-04T18:47:39.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:39.502 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:39.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:39.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.153603_712.1 2024-03-04T18:47:44.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:44.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:44.504 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:44.551 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:44.551 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:44.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:44.614+0000 7f6470e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:44.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:44.615+0000 7f6470e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:44.641 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:44Z"} 2024-03-04T18:47:44.641 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:44Z"} 2024-03-04T18:47:44.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:44.641 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:44.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:44.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:47:49.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:49.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:49.643 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:49.690 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:49.690 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:49.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:49.744+0000 7f7670844800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:49.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:49.745+0000 7f7670844800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.301088_735.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.301088Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:49Z"} 2024-03-04T18:47:49.767 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.301088_735.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.301088Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:49Z"} 2024-03-04T18:47:49.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.301088_735.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.301088Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:49.767 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:49.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.301088_735.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:49.767 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.301088_735.1 2024-03-04T18:47:54.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:54.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:54.769 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:54.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:54.815 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:47:54.870 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:54.871+0000 7f845b00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:54.871 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:47:54.871+0000 7f845b00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:47:54.893 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:54Z"} 2024-03-04T18:47:54.893 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:54Z"} 2024-03-04T18:47:54.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:47:54.894 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:54.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:47:54.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:47:59.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:59.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:47:59.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:47:59.945 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:47:59.945 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:00.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:00.013+0000 7fae2c30d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:00.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:00.014+0000 7fae2c30d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:00.048 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:00Z"} 2024-03-04T18:48:00.048 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:00Z"} 2024-03-04T18:48:00.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:00.048 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:00.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:00.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T18:48:05.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:05.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:05.050 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:05.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:05.103 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:05.177 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:05.177+0000 7f759ad2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:05.178 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:05.178+0000 7f759ad2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:05.210 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:05Z"} 2024-03-04T18:48:05.210 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:05Z"} 2024-03-04T18:48:05.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:05.210 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:05.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:05.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:48:10.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:10.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:10.213 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:10.261 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:10.261 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:10.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:10.311+0000 7f13e2533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:10.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:10.312+0000 7f13e2533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:10.334 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:10Z"} 2024-03-04T18:48:10.334 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:10Z"} 2024-03-04T18:48:10.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:10.335 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:10.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:10.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:48:15.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:15.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:15.336 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:15.383 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:15.384 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:15.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:15.442+0000 7fe0ac50b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:15.442 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:15.443+0000 7fe0ac50b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:15.476 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:15Z"} 2024-03-04T18:48:15.476 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:15Z"} 2024-03-04T18:48:15.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:15.476 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:15.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:15.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:48:20.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:20.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:20.479 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:20.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:20.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:20.581 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:20.582+0000 7f1ed511f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:20.582 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:20.583+0000 7f1ed511f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:20.607 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:20Z"} 2024-03-04T18:48:20.607 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:20Z"} 2024-03-04T18:48:20.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:20.607 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:20.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:20.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:48:25.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:25.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:25.610 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:25.660 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:25.660 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:25.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:25.713+0000 7f15a671f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:25.714 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:25.714+0000 7f15a671f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:25.741 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:25Z"} 2024-03-04T18:48:25.741 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:25Z"} 2024-03-04T18:48:25.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:25.742 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:25.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:25.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.121915_709.1 2024-03-04T18:48:30.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:30.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:30.744 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:30.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:30.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:30.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:30.847+0000 7f61c4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:30.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:30.848+0000 7f61c4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:30.870 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:30Z"} 2024-03-04T18:48:30.871 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:30Z"} 2024-03-04T18:48:30.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:30.871 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:30.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:30.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:48:35.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:35.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:35.874 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:35.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:35.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:36.050 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:36.051+0000 7f5bc7622800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:36.052 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:36.052+0000 7f5bc7622800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:36.139 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:36Z"} 2024-03-04T18:48:36.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:36Z"} 2024-03-04T18:48:36.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:36.139 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:36.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:36.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:48:41.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:41.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:41.141 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:41.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:41.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:41.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:41.312+0000 7f8ffa4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:41.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:41.313+0000 7f8ffa4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:41.342 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:41Z"} 2024-03-04T18:48:41.342 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:41Z"} 2024-03-04T18:48:41.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:41.342 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:41.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:41.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:48:46.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:46.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:46.344 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:46.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:46.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:46.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:46.493+0000 7f9080d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:46.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:46.494+0000 7f9080d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:46.519 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:46Z"} 2024-03-04T18:48:46.519 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:46Z"} 2024-03-04T18:48:46.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:46.519 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:46.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:46.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:48:51.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:51.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:51.522 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:51.568 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:51.569 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:51.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:51.638+0000 7fef0b12c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:51.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:51.639+0000 7fef0b12c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:51.665 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:51Z"} 2024-03-04T18:48:51.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:51Z"} 2024-03-04T18:48:51.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:51.666 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:51.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:51.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:48:56.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:56.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:48:56.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:48:56.714 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:48:56.714 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:48:56.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:56.772+0000 7fc1757a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:56.772 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:48:56.773+0000 7fc1757a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:48:56.802 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:56Z"} 2024-03-04T18:48:56.802 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:56Z"} 2024-03-04T18:48:56.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:48:56.802 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:56.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:48:56.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:49:01.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:01.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:01.804 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:01.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:01.852 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:01.910 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:01.911+0000 7fab94131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:01.911 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:01.911+0000 7fab94131800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:01.934 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:01Z"} 2024-03-04T18:49:01.934 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:01Z"} 2024-03-04T18:49:01.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:01.935 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:01.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:01.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:49:06.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:06.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:06.940 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:06.986 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:06.987 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:07.056+0000 7f1d25511800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:07.057 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:07.057+0000 7f1d25511800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:07.080 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:07Z"} 2024-03-04T18:49:07.081 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:07Z"} 2024-03-04T18:49:07.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:07.081 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:07.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:07.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T18:49:12.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:12.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:12.083 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:12.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:12.132 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:12.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:12.212+0000 7f49be533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:12.213+0000 7f49be533800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:12.249 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:12Z"} 2024-03-04T18:49:12.250 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:12Z"} 2024-03-04T18:49:12.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:12.250 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:12.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:12.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:49:17.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:17.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:17.252 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:17.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:17.301 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:17.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:17.357+0000 7f13e0daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:17.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:17.358+0000 7f13e0daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:17.379 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:17Z"} 2024-03-04T18:49:17.380 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:17Z"} 2024-03-04T18:49:17.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:17.380 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:17.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:17.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:49:22.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:22.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:22.382 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:22.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:22.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:22.480 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:22.480+0000 7f28cc12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:22.481 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:22.481+0000 7f28cc12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:22.503 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:22Z"} 2024-03-04T18:49:22.504 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:22Z"} 2024-03-04T18:49:22.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:22.504 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:22.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:22.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:49:27.508 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2024-03-04T18:49:27.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:27.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:27.510 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:27.621 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:49:27.643 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:49:27.644 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:27.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:49:27.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:27.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:27.645 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:27.693 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:27.694 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:27.763 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:27.763+0000 7fb42692d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:27.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:27.764+0000 7fb42692d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:27.795 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:27Z"} 2024-03-04T18:49:27.796 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:27Z"} 2024-03-04T18:49:27.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:27.796 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:27.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:27.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:49:32.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:32.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:32.799 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:32.851 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:32.851 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:32.905 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:32.905+0000 7f8d25f31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:32.906 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:32.906+0000 7f8d25f31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:32.930 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:32Z"} 2024-03-04T18:49:32.931 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:32Z"} 2024-03-04T18:49:32.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:32.931 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:32.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:32.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:49:37.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:37.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:37.934 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:38.018 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:38.018 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:38.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:38.142+0000 7fbddb02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:38.143 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:38.143+0000 7fbddb02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:38.183 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:38Z"} 2024-03-04T18:49:38.184 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:38Z"} 2024-03-04T18:49:38.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:38.184 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:38.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:38.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T18:49:43.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:43.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:43.187 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:43.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:43.302+0000 7ff9c513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:43.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:43.303+0000 7ff9c513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:43.332 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:43Z"} 2024-03-04T18:49:43.333 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:43Z"} 2024-03-04T18:49:43.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:43.333 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:43.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:43.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:49:48.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:48.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:48.335 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:48.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:48.461+0000 7f838aa34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:48.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:48.462+0000 7f838aa34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:48.492 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:48Z"} 2024-03-04T18:49:48.492 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:48Z"} 2024-03-04T18:49:48.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:48.493 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:48.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:48.493 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:49:53.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:53.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:53.494 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:53.543 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:53.543 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:53.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:53.603+0000 7f26d034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:53.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:53.604+0000 7f26d034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:53.631 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:53Z"} 2024-03-04T18:49:53.631 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:53Z"} 2024-03-04T18:49:53.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:53.632 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:53.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:53.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T18:49:58.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:58.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:49:58.634 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:49:58.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:49:58.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:49:58.749 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:58.750+0000 7fce73814800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:58.750 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:49:58.750+0000 7fce73814800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:49:58.774 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:58Z"} 2024-03-04T18:49:58.774 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:58Z"} 2024-03-04T18:49:58.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:49:58.775 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:58.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:49:58.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:50:03.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:03.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:03.776 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:03.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:03.827 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:03.876 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:03.877+0000 7fd58a23a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:03.877 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:03.878+0000 7fd58a23a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:03.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:03Z"} 2024-03-04T18:50:03.902 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:03Z"} 2024-03-04T18:50:03.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:03.903 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:03.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:03.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:50:08.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:08.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:08.905 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:08.954 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:08.955 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:09.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:09.027+0000 7f99d012b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:09.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:09.028+0000 7f99d012b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:09.049 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:09Z"} 2024-03-04T18:50:09.050 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:09Z"} 2024-03-04T18:50:09.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:09.050 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:09.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:09.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:50:14.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:14.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:14.052 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:14.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:14.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:14.173 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:14.173+0000 7f9960a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:14.174 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:14.174+0000 7f9960a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:14.201 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:14Z"} 2024-03-04T18:50:14.201 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:14Z"} 2024-03-04T18:50:14.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:14.201 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:14.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:14.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:50:19.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:19.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:19.204 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:19.253 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:19.253 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:19.320 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:19.320+0000 7f0a7ef24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:19.320 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:19.321+0000 7f0a7ef24800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:19.346 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:19Z"} 2024-03-04T18:50:19.347 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:19Z"} 2024-03-04T18:50:19.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:19.347 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:19.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:19.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:50:24.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:24.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:24.349 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:24.397 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:24.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:24.448 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:24.448+0000 7f8f70644800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:24.448 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:24.449+0000 7f8f70644800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:24.473 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:24Z"} 2024-03-04T18:50:24.474 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:24Z"} 2024-03-04T18:50:24.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:24.474 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:24.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:24.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:50:29.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:29.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:29.476 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:29.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:29.527 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:29.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:29.591+0000 7f5edcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:29.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:29.592+0000 7f5edcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:29.614 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:29Z"} 2024-03-04T18:50:29.614 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:29Z"} 2024-03-04T18:50:29.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:29.615 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:29.615 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:29.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:50:34.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:34.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:34.616 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:34.663 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:34.663 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:34.713 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:34.713+0000 7fc900c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:34.714 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:34.714+0000 7fc900c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:34.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:34Z"} 2024-03-04T18:50:34.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:34Z"} 2024-03-04T18:50:34.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:34.739 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:34.739 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:34.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:50:39.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:39.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:39.741 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:39.794 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:39.794 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:39.862 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:39.862+0000 7f0196415800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:39.862 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:39.863+0000 7f0196415800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:39.894 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:39Z"} 2024-03-04T18:50:39.894 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:39Z"} 2024-03-04T18:50:39.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:39.895 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:39.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:39.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T18:50:44.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:44.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:44.897 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:44.947 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:44.947 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:45.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:45.008+0000 7fa0e4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:45.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:45.008+0000 7fa0e4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:45.036 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:45Z"} 2024-03-04T18:50:45.036 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:45Z"} 2024-03-04T18:50:45.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:45.037 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:45.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:45.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:50:50.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:50.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:50.039 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:50.112 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:50.113 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:50.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:50.238+0000 7fac3b612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:50.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:50.239+0000 7fac3b612800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:50.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:50Z"} 2024-03-04T18:50:50.265 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:50Z"} 2024-03-04T18:50:50.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:50.265 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:50.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:50.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:50:55.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:55.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:50:55.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:50:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:50:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:50:55.388 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:55.388+0000 7f2e0f20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:55.388 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:50:55.389+0000 7f2e0f20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:50:55.412 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:55Z"} 2024-03-04T18:50:55.412 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:55Z"} 2024-03-04T18:50:55.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:50:55.413 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:55.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:50:55.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:51:00.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:00.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:00.415 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:00.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:00.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:00.519 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:00.519+0000 7ff664d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:00.520 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:00.520+0000 7ff664d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:00.547 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:00Z"} 2024-03-04T18:51:00.547 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:00Z"} 2024-03-04T18:51:00.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:00.547 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:00.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:00.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:51:05.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:05.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:05.550 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:05.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:05.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:05.657 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:05.658+0000 7f3a36285800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:05.658 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:05.659+0000 7f3a36285800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:05.684 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:05Z"} 2024-03-04T18:51:05.684 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:05Z"} 2024-03-04T18:51:05.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:05.685 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:05.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:05.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T18:51:10.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:10.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:10.686 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:10.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:10.803+0000 7f32000c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:10.804 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:10.804+0000 7f32000c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:10.828 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:10Z"} 2024-03-04T18:51:10.829 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:10Z"} 2024-03-04T18:51:10.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:10.829 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:10.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:10.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:51:15.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:15.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:15.831 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:15.878 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:15.878 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:15.935 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:15.936+0000 7f0382331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:15.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:15.937+0000 7f0382331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:15.960 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:15Z"} 2024-03-04T18:51:15.960 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:15Z"} 2024-03-04T18:51:15.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:15.960 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:15.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:15.960 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:51:20.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:20.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:20.963 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:21.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:21.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:21.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:21.059+0000 7fe871711800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:21.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:21.060+0000 7fe871711800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:21.084 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:21Z"} 2024-03-04T18:51:21.084 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:21Z"} 2024-03-04T18:51:21.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:21.085 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:21.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:21.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:51:26.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:26.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:26.087 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:26.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:26.134 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:26.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:26.194+0000 7fd10e442800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:26.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:26.195+0000 7fd10e442800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:26.219 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:26Z"} 2024-03-04T18:51:26.219 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:26Z"} 2024-03-04T18:51:26.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:26.219 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:26.220 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:26.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:51:31.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:31.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:31.221 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:31.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:31.268 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:31.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:31.333+0000 7f258ccc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:31.333 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:31.334+0000 7f258ccc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:31.359 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:31Z"} 2024-03-04T18:51:31.359 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:31Z"} 2024-03-04T18:51:31.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:31.360 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:31.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:31.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:51:36.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:36.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:36.362 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:36.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:36.412 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:36.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:36.477+0000 7efdd072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:36.478 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:36.478+0000 7efdd072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:36.502 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:36Z"} 2024-03-04T18:51:36.503 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:36Z"} 2024-03-04T18:51:36.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:36.503 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:36.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:36.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:51:41.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:41.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:41.505 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:41.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:41.559 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:41.626 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:41.626+0000 7fca9f90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:41.627 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:41.627+0000 7fca9f90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:41.655 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.183700_722.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.183700Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:41Z"} 2024-03-04T18:51:41.655 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.183700_722.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.183700Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:41Z"} 2024-03-04T18:51:41.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.183700_722.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.183700Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:41.656 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:41.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.183700_722.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:41.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.183700_722.1 2024-03-04T18:51:46.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:46.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:46.658 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:46.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:46.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:46.845 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:46.845+0000 7f108f811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:46.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:46.846+0000 7f108f811800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:46.909 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:46Z"} 2024-03-04T18:51:46.909 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:46Z"} 2024-03-04T18:51:46.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:46.909 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:46.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:46.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:51:51.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:51.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:51.911 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:51.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:52.036+0000 7febf9f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:52.037 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:52.037+0000 7febf9f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:52.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:52Z"} 2024-03-04T18:51:52.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:52Z"} 2024-03-04T18:51:52.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:52.065 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:52.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:52.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:51:57.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:57.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:51:57.068 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:51:57.115 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:51:57.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:51:57.172 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:57.172+0000 7feadab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:57.172 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:51:57.173+0000 7feadab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:51:57.195 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:57Z"} 2024-03-04T18:51:57.196 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:57Z"} 2024-03-04T18:51:57.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:51:57.196 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:57.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:51:57.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:52:02.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:02.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:02.199 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:02.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:02.295+0000 7feabefa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:02.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:02.296+0000 7feabefa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:02.325 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.327181_739.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.327181Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:02Z"} 2024-03-04T18:52:02.326 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.327181_739.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.327181Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:02Z"} 2024-03-04T18:52:02.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.327181_739.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.327181Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:02.326 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:02.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.327181_739.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:02.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.327181_739.1 2024-03-04T18:52:07.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:07.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:07.328 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:07.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:07.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:07.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:07.444+0000 7f7dd4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:07.445+0000 7f7dd4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:07.470 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:07Z"} 2024-03-04T18:52:07.470 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:07Z"} 2024-03-04T18:52:07.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:07.471 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:07.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:07.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:52:12.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:12.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:12.472 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:12.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:12.519 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:12.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:12.568+0000 7fa513285800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:12.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:12.569+0000 7fa513285800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:12.592 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:12Z"} 2024-03-04T18:52:12.593 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:12Z"} 2024-03-04T18:52:12.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:12.593 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:12.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:12.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T18:52:17.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:17.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:17.595 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:17.644 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:17.644 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:17.700 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:17.700+0000 7fdd4e038800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:17.701 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:17.701+0000 7fdd4e038800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:17.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:17Z"} 2024-03-04T18:52:17.729 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:17Z"} 2024-03-04T18:52:17.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:17.729 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:17.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:17.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:52:22.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:22.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:22.732 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:22.782 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:22.782 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:22.836+0000 7ff62b528800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:22.837+0000 7ff62b528800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:22.861 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:22Z"} 2024-03-04T18:52:22.861 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:22Z"} 2024-03-04T18:52:22.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:22.862 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:22.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:22.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T18:52:27.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:27.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:27.864 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:27.964+0000 7f673ac30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:27.964 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:27.965+0000 7f673ac30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:27.995 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:27Z"} 2024-03-04T18:52:27.995 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:27Z"} 2024-03-04T18:52:27.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:27.996 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:27.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:27.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:52:32.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:32.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:32.998 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:33.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:33.122+0000 7f58d4815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:33.122 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:33.123+0000 7f58d4815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:33.153 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:33Z"} 2024-03-04T18:52:33.154 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:33Z"} 2024-03-04T18:52:33.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:33.154 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:33.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:33.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:52:38.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:38.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:38.157 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:38.210 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:38.211 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:38.265+0000 7f24d8f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:38.266+0000 7f24d8f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:38.290 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:38Z"} 2024-03-04T18:52:38.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:38Z"} 2024-03-04T18:52:38.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:38.291 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:38.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:38.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:52:43.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:43.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:43.292 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:43.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:43.346 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:43.401 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:43.401+0000 7f3a15331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:43.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:43.402+0000 7f3a15331800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:43.425 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:43Z"} 2024-03-04T18:52:43.425 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:43Z"} 2024-03-04T18:52:43.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:43.426 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:43.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:43.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:52:48.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:48.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:48.427 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:48.480 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:48.480 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:48.582 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:48.583+0000 7fb60df2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:48.595 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:48.594+0000 7fb60df2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:48.685 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:48Z"} 2024-03-04T18:52:48.685 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:48Z"} 2024-03-04T18:52:48.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:48.686 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:48.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:48.686 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:52:53.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:53.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:53.688 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:53.787 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:53.787+0000 7f5cff92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:53.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:53.788+0000 7f5cff92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:53.813 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.223764_728.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.223764Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:53Z"} 2024-03-04T18:52:53.813 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.223764_728.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.223764Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:53Z"} 2024-03-04T18:52:53.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.223764_728.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.223764Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:53.814 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:53.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.223764_728.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:53.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.223764_728.1 2024-03-04T18:52:58.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:58.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:52:58.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:52:58.907 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:52:58.908 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:52:59.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:59.024+0000 7f4f0b824800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:59.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:52:59.024+0000 7f4f0b824800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:52:59.054 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:59Z"} 2024-03-04T18:52:59.054 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:59Z"} 2024-03-04T18:52:59.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:52:59.055 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:59.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:52:59.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:53:04.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:04.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:04.057 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:04.106 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:04.106 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:04.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:04.157+0000 7fae62614800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:04.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:04.158+0000 7fae62614800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:04Z"} 2024-03-04T18:53:04.182 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:04Z"} 2024-03-04T18:53:04.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:04.182 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:04.182 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:04.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:53:09.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:09.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:09.184 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:09.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:09.242 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:09.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:09.305+0000 7f14eeb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:09.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:09.306+0000 7f14eeb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:09.330 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:09Z"} 2024-03-04T18:53:09.331 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:09Z"} 2024-03-04T18:53:09.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:09.331 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:09.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:09.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:53:14.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:14.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:14.333 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:14.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:14.429+0000 7f64c4f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:14.429 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:14.430+0000 7f64c4f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:14Z"} 2024-03-04T18:53:14.466 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:14Z"} 2024-03-04T18:53:14.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:14.466 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:14.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:14.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:53:19.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:19.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:19.468 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:19.514 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:19.514 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:19.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:19.562+0000 7f9c6a444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:19.563+0000 7f9c6a444800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:19.587 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:19Z"} 2024-03-04T18:53:19.587 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:19Z"} 2024-03-04T18:53:19.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:19.588 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:19.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:19.588 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:53:24.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:24.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:24.590 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:24.637 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:24.637 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:24.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:24.708+0000 7fb7cc511800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:24.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:24.709+0000 7fb7cc511800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:24.732 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:24Z"} 2024-03-04T18:53:24.732 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:24Z"} 2024-03-04T18:53:24.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:24.733 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:24.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:24.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:53:29.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:29.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:29.735 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:29.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:29.781 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:29.855 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:29.855+0000 7fc08408c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:29.855 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:29.856+0000 7fc08408c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:29.888 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:29Z"} 2024-03-04T18:53:29.888 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:29Z"} 2024-03-04T18:53:29.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:29.889 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:29.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:29.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T18:53:34.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:34.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:34.892 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:34.939 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:34.939 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:35.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:35.015+0000 7f4923f0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:35.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:35.016+0000 7f4923f0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:35.049 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:35Z"} 2024-03-04T18:53:35.049 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:35Z"} 2024-03-04T18:53:35.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:35.049 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:35.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:35.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.242177_729.1 2024-03-04T18:53:40.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:40.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:40.051 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:40.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:40.098 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:40.173 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:40.173+0000 7fe2fc431800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:40.174 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:40.174+0000 7fe2fc431800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:40.210 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:40Z"} 2024-03-04T18:53:40.210 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:40Z"} 2024-03-04T18:53:40.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:40.210 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:40.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:40.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:53:45.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:45.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:45.212 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:45.265 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:45.266 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:45.335 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:45.335+0000 7fa751e2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:45.336 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:45.336+0000 7fa751e2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:45.362 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:45Z"} 2024-03-04T18:53:45.362 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:45Z"} 2024-03-04T18:53:45.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:45.363 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:45.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:45.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:53:50.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:50.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:50.365 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:50.411 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:50.411 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:50.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:50.572+0000 7fe321333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:50.572 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:50.573+0000 7fe321333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:50.596 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:50Z"} 2024-03-04T18:53:50.596 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:50Z"} 2024-03-04T18:53:50.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:50.597 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:50.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:50.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T18:53:55.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:55.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:53:55.599 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:53:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:53:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:53:55.701 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:55.701+0000 7fdc10733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:55.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:53:55.702+0000 7fdc10733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:53:55.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:55Z"} 2024-03-04T18:53:55.724 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:55Z"} 2024-03-04T18:53:55.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:53:55.725 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:55.725 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:53:55.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:54:00.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:00.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:00.726 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:00.831 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:00.831+0000 7f3e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:00.832 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:00.832+0000 7f3e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:00.855 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:00Z"} 2024-03-04T18:54:00.855 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:00Z"} 2024-03-04T18:54:00.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:00.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:00.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:00.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:54:05.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:05.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:05.856 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:05.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:05.906 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:05.962 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:05.962+0000 7f854f812800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:05.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:05.963+0000 7f854f812800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:05.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:05Z"} 2024-03-04T18:54:05.986 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:05Z"} 2024-03-04T18:54:05.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:05.986 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:05.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:05.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:54:10.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:10.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:10.989 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:11.041 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:11.041 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:11.109 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:11.109+0000 7ff225211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:11.110 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:11.110+0000 7ff225211800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:11.133 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:11Z"} 2024-03-04T18:54:11.133 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:11Z"} 2024-03-04T18:54:11.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:11.133 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:11.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:11.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:54:16.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:16.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:16.135 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:16.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:16.188 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:16.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:16.342+0000 7f72dfa3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:16.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:16.347+0000 7f72dfa3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:16.410 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:16Z"} 2024-03-04T18:54:16.410 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:16Z"} 2024-03-04T18:54:16.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:16.411 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:16.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:16.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:54:21.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:21.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:21.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:21.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:21.624+0000 7f925aa13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:21.630 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:21.630+0000 7f925aa13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:21.684 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:21Z"} 2024-03-04T18:54:21.685 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:21Z"} 2024-03-04T18:54:21.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:21.685 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:21.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:21.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:54:26.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:26.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:26.686 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:26.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:26.732 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:26.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:26.797+0000 7f6d44933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:26.798 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:26.798+0000 7f6d44933800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:26.822 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:26Z"} 2024-03-04T18:54:26.823 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:26Z"} 2024-03-04T18:54:26.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:26.823 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:26.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:26.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:54:31.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:31.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:31.825 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:31.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:31.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:31.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:31.928+0000 7f555a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:31.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:31.929+0000 7f555a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:31.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:31Z"} 2024-03-04T18:54:31.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:31Z"} 2024-03-04T18:54:31.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:31.959 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:31.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:31.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:54:36.962 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-03-04T18:54:36.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:36.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:36.964 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:37.072 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:54:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:54:37.093 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:37.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:54:37.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:37.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:37.094 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:37.142 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:37.142 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:37.210 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:37.210+0000 7f50fe024800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:37.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:37.211+0000 7f50fe024800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:37.233 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:37Z"} 2024-03-04T18:54:37.234 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:37Z"} 2024-03-04T18:54:37.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:37.234 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:37.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:37.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:54:42.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:42.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:42.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:42.283 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:42.284 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:42.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:42.346+0000 7fbd02a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:42.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:42.347+0000 7fbd02a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:42.376 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:42Z"} 2024-03-04T18:54:42.376 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:42Z"} 2024-03-04T18:54:42.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:42.376 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:42.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:42.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:54:47.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:47.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:47.378 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:47.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:47.426 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:47.482 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:47.484+0000 7f84a9333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:47.483 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:47.485+0000 7f84a9333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:47.511 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:47Z"} 2024-03-04T18:54:47.511 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:47Z"} 2024-03-04T18:54:47.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:47.512 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:47.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:47.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:54:52.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:52.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:52.514 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:52.568 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:52.568 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:52.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:52.637+0000 7f79bf013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:52.638 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:52.638+0000 7f79bf013800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:52.662 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:52Z"} 2024-03-04T18:54:52.662 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:52Z"} 2024-03-04T18:54:52.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:52.663 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:52.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:52.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:54:57.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:57.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:54:57.664 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:54:57.714 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:54:57.714 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:57.777+0000 7f2b4b333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:57.778 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:54:57.778+0000 7f2b4b333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:54:57.802 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:57Z"} 2024-03-04T18:54:57.803 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:57Z"} 2024-03-04T18:54:57.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:54:57.803 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:57.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:54:57.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:55:02.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:02.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:02.805 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:02.858 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:02.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:02.919+0000 7fd319c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:02.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:02.920+0000 7fd319c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:02.952 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:02Z"} 2024-03-04T18:55:02.952 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:02Z"} 2024-03-04T18:55:02.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:02.952 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:02.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:02.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:55:07.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:07.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:07.954 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:08.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:08.005 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:08.075 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:08.075+0000 7f6b88044800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:08.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:08.076+0000 7f6b88044800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:08.112 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:08Z"} 2024-03-04T18:55:08.112 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:08Z"} 2024-03-04T18:55:08.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:08.113 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:08.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:08.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T18:55:13.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:13.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:13.114 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:13.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:13.184 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:13.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:13.294+0000 7f0f30616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:13.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:13.295+0000 7f0f30616800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:13.318 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:13Z"} 2024-03-04T18:55:13.318 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:13Z"} 2024-03-04T18:55:13.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:13.318 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:13.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:13.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:55:18.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:18.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:18.320 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:18.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:18.369 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:18.425 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:18.426+0000 7ff0ed088800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:18.426 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:18.427+0000 7ff0ed088800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:18.449 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:18Z"} 2024-03-04T18:55:18.449 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:18Z"} 2024-03-04T18:55:18.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:18.450 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:18.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:18.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:55:23.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:23.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:23.451 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:23.497 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:23.498 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:23.547 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:23.547+0000 7fd317e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:23.548 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:23.548+0000 7fd317e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:23.572 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:23Z"} 2024-03-04T18:55:23.572 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:23Z"} 2024-03-04T18:55:23.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:23.572 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:23.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:23.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:55:28.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:28.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:28.574 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:28.622 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:28.623 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:28.671 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:28.672+0000 7ff136c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:28.672 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:28.673+0000 7ff136c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:28.696 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:28Z"} 2024-03-04T18:55:28.696 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:28Z"} 2024-03-04T18:55:28.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:28.697 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:28.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:28.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:55:33.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:33.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:33.698 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:33.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:33.745 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:33.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:33.796+0000 7fa5ea88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:33.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:33.797+0000 7fa5ea88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:33.821 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:33Z"} 2024-03-04T18:55:33.822 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:33Z"} 2024-03-04T18:55:33.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:33.822 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:33.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:33.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:55:38.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:38.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:38.824 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:38.873 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:38.873 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:38.928 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:38.929+0000 7f308aa44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:38.929 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:38.930+0000 7f308aa44800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:38.954 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:38Z"} 2024-03-04T18:55:38.954 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:38Z"} 2024-03-04T18:55:38.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:38.954 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:38.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:38.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:55:43.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:43.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:43.956 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:44.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:44.010 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:44.066 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:44.066+0000 7f92e510f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:44.067 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:44.067+0000 7f92e510f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:44.092 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:44Z"} 2024-03-04T18:55:44.093 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:44Z"} 2024-03-04T18:55:44.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:44.093 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:44.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:44.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.002120_706.1 2024-03-04T18:55:49.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:49.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:49.095 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:49.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:49.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:49.209 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:49.209+0000 7f0d5aa0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:49.210 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:49.210+0000 7f0d5aa0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:49.233 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:49Z"} 2024-03-04T18:55:49.233 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:49Z"} 2024-03-04T18:55:49.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:49.234 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:49.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:49.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T18:55:54.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:54.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:54.235 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:54.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:54.289 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:54.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:54.345+0000 7f8490a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:54.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:54.345+0000 7f8490a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:54.370 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:54Z"} 2024-03-04T18:55:54.370 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:54Z"} 2024-03-04T18:55:54.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:54.370 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:54.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:54.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:55:59.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:59.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:55:59.372 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:55:59.452 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:55:59.452 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:55:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:59.584+0000 7fac4b424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:59.584 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:55:59.585+0000 7fac4b424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:55:59.609 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:59Z"} 2024-03-04T18:55:59.610 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:59Z"} 2024-03-04T18:55:59.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:55:59.610 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:59.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:55:59.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:56:04.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:04.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:04.612 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:04.658 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:04.658 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:04.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:04.712+0000 7f5a0982e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:04.712 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:04.712+0000 7f5a0982e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:04.735 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:04Z"} 2024-03-04T18:56:04.735 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:04Z"} 2024-03-04T18:56:04.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:04.736 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:04.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:04.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:56:09.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:09.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:09.737 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:09.797 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:09.798 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:09.849 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:09.850+0000 7fe9917a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:09.850 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:09.851+0000 7fe9917a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:09.874 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:09Z"} 2024-03-04T18:56:09.874 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:09Z"} 2024-03-04T18:56:09.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:09.875 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:09.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:09.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:56:14.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:14.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:14.876 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:15.000 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:15.000+0000 7f1224fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:15.001 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:15.001+0000 7f1224fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:15.023 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:15Z"} 2024-03-04T18:56:15.023 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:15Z"} 2024-03-04T18:56:15.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:15.024 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:15.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:15.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:56:20.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:20.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:20.026 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:20.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:20.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:20.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:20.121+0000 7f11eb42f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:20.121 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:20.122+0000 7f11eb42f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:20.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:20Z"} 2024-03-04T18:56:20.145 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:20Z"} 2024-03-04T18:56:20.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:20.145 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:20.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:20.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:56:25.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:25.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:25.147 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:25.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:25.193 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:25.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:25.254+0000 7f580da2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:25.255 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:25.255+0000 7f580da2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:25.278 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:25Z"} 2024-03-04T18:56:25.278 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:25Z"} 2024-03-04T18:56:25.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:25.279 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:25.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:25.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:56:30.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:30.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:30.281 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:30.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:30.328 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:30.378 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:30.379+0000 7fe85bb3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:30.379 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:30.379+0000 7fe85bb3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:30.404 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:30Z"} 2024-03-04T18:56:30.405 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:30Z"} 2024-03-04T18:56:30.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:30.405 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:30.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:30.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:56:35.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:35.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:35.408 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:35.458 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:35.458 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:35.521 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:35.521+0000 7f7019c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:35.522 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:35.522+0000 7f7019c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:35.547 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:35Z"} 2024-03-04T18:56:35.547 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:35Z"} 2024-03-04T18:56:35.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:35.548 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:35.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:35.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:56:40.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:40.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:40.550 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:40.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:40.599 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:40.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:40.660+0000 7fecd3111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:40.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:40.661+0000 7fecd3111800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:40.689 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:40Z"} 2024-03-04T18:56:40.690 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:40Z"} 2024-03-04T18:56:40.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:40.690 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:40.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:40.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T18:56:45.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:45.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:45.692 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:45.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:45.739 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:45.792+0000 7f3ac1638800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:45.793+0000 7f3ac1638800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:45.821 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:45Z"} 2024-03-04T18:56:45.821 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:45Z"} 2024-03-04T18:56:45.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:45.821 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:45.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:45.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:56:50.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:50.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:50.823 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:50.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:50.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:50.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:50.932+0000 7f37f5810800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:50.932 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:50.933+0000 7f37f5810800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:50.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:50Z"} 2024-03-04T18:56:50.958 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:50Z"} 2024-03-04T18:56:50.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:50.959 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:50.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:50.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T18:56:55.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:55.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:56:55.960 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:56:56.012 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:56:56.013 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:56:56.067 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:56.067+0000 7f4efd067800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:56.068 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:56:56.068+0000 7f4efd067800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:56:56.094 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:56Z"} 2024-03-04T18:56:56.094 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:56Z"} 2024-03-04T18:56:56.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:56:56.095 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:56.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:56:56.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T18:57:01.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:01.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:01.096 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:01.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:01.146 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:01.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:01.212+0000 7f755c615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:01.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:01.213+0000 7f755c615800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:01.238 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:01Z"} 2024-03-04T18:57:01.238 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:01Z"} 2024-03-04T18:57:01.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:01.238 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:01.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:01.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:57:06.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:06.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:06.240 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:06.355+0000 7fc767630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:06.356+0000 7fc767630800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:06.388 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:06Z"} 2024-03-04T18:57:06.388 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:06Z"} 2024-03-04T18:57:06.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:06.388 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:06.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:06.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T18:57:11.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:11.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:11.390 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:11.592 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:11.592+0000 7fec9a1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:11.592 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:11.592+0000 7fec9a1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:11.617 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:11Z"} 2024-03-04T18:57:11.617 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:11Z"} 2024-03-04T18:57:11.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:11.617 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:11.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:11.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:57:16.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:16.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:16.619 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:16.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:16.666 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:16.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:16.719+0000 7f379df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:16.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:16.720+0000 7f379df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:16.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:16Z"} 2024-03-04T18:57:16.747 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:16Z"} 2024-03-04T18:57:16.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:16.747 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:16.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:16.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T18:57:21.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:21.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:21.748 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:21.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:21.796 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:21.842 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:21.843+0000 7fe3fe214800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:21.843 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:21.843+0000 7fe3fe214800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:21.865 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:21Z"} 2024-03-04T18:57:21.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:21Z"} 2024-03-04T18:57:21.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:21.866 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:21.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:21.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:57:26.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:26.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:26.867 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:26.917 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:26.917 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:26.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:26.974+0000 7efea3617800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:26.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:26.975+0000 7efea3617800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:26.998 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:26Z"} 2024-03-04T18:57:26.998 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:26Z"} 2024-03-04T18:57:26.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:26.998 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:26.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:26.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:57:31.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:32.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:32.000 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:32.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:32.052 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:32.119 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:32.121+0000 7f56f2e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:32.120 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:32.122+0000 7f56f2e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:32.153 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:32Z"} 2024-03-04T18:57:32.154 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:32Z"} 2024-03-04T18:57:32.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:32.154 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:32.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:32.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:57:37.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:37.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:37.155 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:37.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:37.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:37.282+0000 7f5a6e624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:37.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:37.283+0000 7f5a6e624800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:37.307 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:37Z"} 2024-03-04T18:57:37.307 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:37Z"} 2024-03-04T18:57:37.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:37.307 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:37.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:37.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T18:57:42.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:42.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:42.309 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:42.383 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:42.383 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:42.524+0000 7efefe133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:42.525+0000 7efefe133800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:42.560 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:42Z"} 2024-03-04T18:57:42.560 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:42Z"} 2024-03-04T18:57:42.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:42.560 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:42.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:42.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:57:47.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:47.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:47.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:47.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:47.620 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:47.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:47.689+0000 7f3c4a333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:47.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:47.690+0000 7f3c4a333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:47.715 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:47Z"} 2024-03-04T18:57:47.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:47Z"} 2024-03-04T18:57:47.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.214996_723.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.214996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:47.715 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:47.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.214996_723.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:47.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.214996_723.1 2024-03-04T18:57:52.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:52.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:52.717 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:52.821 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:52.821+0000 7f0e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:52.822+0000 7f0e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:52.854 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:52Z"} 2024-03-04T18:57:52.854 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:52Z"} 2024-03-04T18:57:52.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:52.855 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:52.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:52.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:57:57.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:57.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:57:57.856 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:57:57.903 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:57:57.903 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:57:57.964 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:57.965+0000 7fa29b733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:57.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:57:57.966+0000 7fa29b733800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:57:57.988 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:57Z"} 2024-03-04T18:57:57.988 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:57Z"} 2024-03-04T18:57:57.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:57:57.989 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:57.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:57:57.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:58:02.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:02.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:02.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:03.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:03.037 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:03.096+0000 7fe9f1467800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:03.097+0000 7fe9f1467800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:03.124 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:03Z"} 2024-03-04T18:58:03.124 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:03Z"} 2024-03-04T18:58:03.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:03.125 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:03.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:03.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T18:58:08.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:08.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:08.127 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:08.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:08.174 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:08.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:08.222+0000 7f32fa415800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:08.223+0000 7f32fa415800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:08.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:08Z"} 2024-03-04T18:58:08.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:08Z"} 2024-03-04T18:58:08.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:08.246 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:08.247 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:08.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.125073_710.1 2024-03-04T18:58:13.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:13.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:13.248 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:13.295 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:13.295 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:13.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:13.365+0000 7fe9fd82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:13.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:13.366+0000 7fe9fd82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:13.398 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:13Z"} 2024-03-04T18:58:13.398 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:13Z"} 2024-03-04T18:58:13.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:13.398 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:13.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:13.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:58:18.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:18.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:18.400 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:18.446 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:18.447 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:18.508 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:18.508+0000 7f05b861b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:18.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:18.509+0000 7f05b861b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:18.533 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:18Z"} 2024-03-04T18:58:18.533 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:18Z"} 2024-03-04T18:58:18.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:18.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:18.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:18.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:58:23.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:23.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:23.535 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:23.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:23.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:23.643 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:23.643+0000 7fcfe333c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:23.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:23.644+0000 7fcfe333c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:23.672 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:23Z"} 2024-03-04T18:58:23.672 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:23Z"} 2024-03-04T18:58:23.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:23.672 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:23.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:23.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:58:28.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:28.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:28.674 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:28.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:28.724 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:28.793 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:28.794+0000 7f112e414800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:28.794 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:28.795+0000 7f112e414800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:28.823 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:28Z"} 2024-03-04T18:58:28.824 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:28Z"} 2024-03-04T18:58:28.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:28.824 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:28.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:28.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:58:33.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:33.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:33.826 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:33.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:33.880 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:33.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:33.950+0000 7f7891a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:33.950 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:33.951+0000 7f7891a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:33.974 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:33Z"} 2024-03-04T18:58:33.974 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:33Z"} 2024-03-04T18:58:33.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:33.975 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:33.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:33.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:58:38.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:38.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:38.977 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:39.030 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:39.030 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:39.107 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:39.107+0000 7fa78652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:39.108 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:39.108+0000 7fa78652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:39.134 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:39Z"} 2024-03-04T18:58:39.134 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:39Z"} 2024-03-04T18:58:39.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:39.134 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:39.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:39.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:58:44.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:44.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:44.139 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:44.194 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:44.194 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:44.255 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:44.256+0000 7f4200333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:44.256+0000 7f4200333800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:44.280 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571979.256260_731.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:19.256260Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:44Z"} 2024-03-04T18:58:44.280 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571979.256260_731.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:19.256260Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:44Z"} 2024-03-04T18:58:44.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571979.256260_731.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:19.256260Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:44.281 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:44.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571979.256260_731.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:44.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571979.256260_731.1 2024-03-04T18:58:49.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:49.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:49.285 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:49.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:49.339 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:49.506 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:49.506+0000 7fc690815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:49.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:49.509+0000 7fc690815800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:49.534 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:49Z"} 2024-03-04T18:58:49.534 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:49Z"} 2024-03-04T18:58:49.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:49.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:49.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:49.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:58:54.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:54.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:54.537 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:54.644 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:54.645+0000 7f88a9212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:54.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:54.645+0000 7f88a9212800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:54.672 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:54Z"} 2024-03-04T18:58:54.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:54Z"} 2024-03-04T18:58:54.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:54.673 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:54.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:54.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T18:58:59.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:59.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:58:59.674 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:58:59.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:58:59.736 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:58:59.793 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:59.794+0000 7f6b1ce13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:59.794 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:58:59.794+0000 7f6b1ce13800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:58:59.822 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:59Z"} 2024-03-04T18:58:59.822 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:59Z"} 2024-03-04T18:58:59.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:58:59.823 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:59.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:58:59.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T18:59:04.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:04.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:04.825 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:04.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:04.872 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:04.944 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:04.945+0000 7f7e4662d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:04.945 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:04.946+0000 7f7e4662d800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:04.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:04Z"} 2024-03-04T18:59:04.971 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:04Z"} 2024-03-04T18:59:04.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:04.971 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:04.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:04.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T18:59:09.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:09.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:09.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:10.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:10.074 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:10.074+0000 7fb73bc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:10.075 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:10.075+0000 7fb73bc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:10.097 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:10Z"} 2024-03-04T18:59:10.097 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:10Z"} 2024-03-04T18:59:10.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:10.098 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:10.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:10.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T18:59:15.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:15.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:15.099 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:15.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:15.150 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:15.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:15.213+0000 7efee2213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:15.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:15.214+0000 7efee2213800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:15.242 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:15Z"} 2024-03-04T18:59:15.243 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:15Z"} 2024-03-04T18:59:15.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:15.243 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:15.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:15.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:59:20.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:20.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:20.245 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:20.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:20.293 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:20.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:20.352+0000 7f2c09130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:20.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:20.353+0000 7f2c09130800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:20.376 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:20Z"} 2024-03-04T18:59:20.376 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:20Z"} 2024-03-04T18:59:20.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:20.377 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:20.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:20.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T18:59:25.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:25.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:25.378 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:25.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:25.430 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:25.495 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:25.496+0000 7f02df32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:25.496 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:25.497+0000 7f02df32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:25.525 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:25Z"} 2024-03-04T18:59:25.525 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:25Z"} 2024-03-04T18:59:25.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:25.526 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:25.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:25.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T18:59:30.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:30.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:30.527 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:30.577 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:30.577 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:30.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:30.634+0000 7f91d428a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:30.634 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:30.635+0000 7f91d428a800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:30.659 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:30Z"} 2024-03-04T18:59:30.660 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:30Z"} 2024-03-04T18:59:30.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:30.660 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:30.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:30.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T18:59:35.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:35.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:35.661 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:35.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:35.711 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:35.771 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:35.772+0000 7ff45013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:35.774 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:35.774+0000 7ff45013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:35.805 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:35Z"} 2024-03-04T18:59:35.805 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:35Z"} 2024-03-04T18:59:35.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:35.806 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:35.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:35.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T18:59:40.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:40.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:40.807 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:40.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:40.918+0000 7f2c9f244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:40.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:40.919+0000 7f2c9f244800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:40.947 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:40Z"} 2024-03-04T18:59:40.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:40Z"} 2024-03-04T18:59:40.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:40.948 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:40.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:40.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:59:45.950 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... FAIL 2024-03-04T18:59:45.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:45.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:45.952 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:46.337 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:46.476 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T18:59:46.604 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T18:59:46.605 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:46.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T18:59:46.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:46.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:46.605 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:46.652 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:46.652 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:46.707+0000 7fd63ff31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:46.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:46.708+0000 7fd63ff31800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:46.733 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:46Z"} 2024-03-04T18:59:46.733 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:46Z"} 2024-03-04T18:59:46.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:46.733 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:46.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:46.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T18:59:51.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:51.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:51.735 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:51.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:51.795 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:51.846 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:51.846+0000 7fc237f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:51.847 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:51.847+0000 7fc237f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:51.870 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:51Z"} 2024-03-04T18:59:51.870 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:51Z"} 2024-03-04T18:59:51.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:51.870 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:51.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:51.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T18:59:56.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:56.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T18:59:56.872 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T18:59:56.926 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T18:59:56.926 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T18:59:56.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:56.980+0000 7f8c153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:56.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T18:59:56.981+0000 7f8c153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T18:59:57.010 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:56Z"} 2024-03-04T18:59:57.010 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:56Z"} 2024-03-04T18:59:57.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T18:59:57.011 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:57.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T18:59:57.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:00:02.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:02.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:02.013 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:02.060 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:02.061 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:02.112+0000 7f1a4c80f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:02.112 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:02.113+0000 7f1a4c80f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:02.136 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:02Z"} 2024-03-04T19:00:02.136 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:02Z"} 2024-03-04T19:00:02.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:02.137 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:02.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:02.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.242177_729.1 2024-03-04T19:00:07.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:07.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:07.138 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:07.251 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:07.251+0000 7f19fac24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:07.252 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:07.252+0000 7f19fac24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:07.276 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:07Z"} 2024-03-04T19:00:07.276 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:07Z"} 2024-03-04T19:00:07.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:07.276 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:07.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:07.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:00:12.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:12.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:12.278 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:12.355 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:12.356 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:12.483 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:12.483+0000 7f85d0a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:12.484 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:12.484+0000 7f85d0a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:12.517 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:12Z"} 2024-03-04T19:00:12.517 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:12Z"} 2024-03-04T19:00:12.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:12.517 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:12.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:12.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:00:17.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:17.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:17.519 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:17.566 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:17.566 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:17.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:17.622+0000 7fe91509d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:17.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:17.623+0000 7fe91509d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:17.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:17Z"} 2024-03-04T19:00:17.646 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:17Z"} 2024-03-04T19:00:17.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:17.646 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:17.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:17.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:00:22.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:22.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:22.648 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:22.694 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:22.694 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:22.765 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:22.765+0000 7f67c6f0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:22.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:22.766+0000 7f67c6f0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:22.798 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:22Z"} 2024-03-04T19:00:22.798 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:22Z"} 2024-03-04T19:00:22.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:22.799 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:22.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:22.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:00:27.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:27.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:27.801 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:27.847 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:27.848 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:27.917 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:27.917+0000 7f2c52230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:27.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:27.918+0000 7f2c52230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:27.945 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:27Z"} 2024-03-04T19:00:27.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:27Z"} 2024-03-04T19:00:27.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:27.946 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:27.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:27.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:00:32.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:32.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:32.947 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:32.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:32.998 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:33.050+0000 7f49bff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:33.051+0000 7f49bff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:33.075 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:33Z"} 2024-03-04T19:00:33.076 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:33Z"} 2024-03-04T19:00:33.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:33.076 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:33.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:33.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.153603_712.1 2024-03-04T19:00:38.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:38.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:38.078 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:38.126 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:38.126 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:38.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:38.185+0000 7f6a9908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:38.186+0000 7f6a9908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:38.212 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.351182_743.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.351182Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:38Z"} 2024-03-04T19:00:38.212 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.351182_743.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.351182Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:38Z"} 2024-03-04T19:00:38.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.351182_743.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.351182Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:38.213 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:38.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.351182_743.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:38.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.351182_743.1 2024-03-04T19:00:43.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:43.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:43.214 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:43.263 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:43.263 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:43.324 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:43.324+0000 7f6605d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:43.325 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:43.325+0000 7f6605d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:43.348 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:43Z"} 2024-03-04T19:00:43.348 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:43Z"} 2024-03-04T19:00:43.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:43.349 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:43.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:43.349 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:00:48.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:48.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:48.350 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:48.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:48.459+0000 7f1511a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:48.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:48.460+0000 7f1511a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:48.488 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:48Z"} 2024-03-04T19:00:48.488 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:48Z"} 2024-03-04T19:00:48.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:48.488 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:48.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:48.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:00:53.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:53.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:53.491 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:53.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:53.540 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:53.596 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:53.596+0000 7f969ad24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:53.597 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:53.597+0000 7f969ad24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:53.627 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:53Z"} 2024-03-04T19:00:53.627 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:53Z"} 2024-03-04T19:00:53.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:53.628 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:53.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:53.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:00:58.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:58.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:00:58.630 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:00:58.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:00:58.683 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:00:58.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:58.746+0000 7f72d0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:58.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:00:58.748+0000 7f72d0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:00:58.771 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:58Z"} 2024-03-04T19:00:58.771 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:58Z"} 2024-03-04T19:00:58.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:00:58.772 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:58.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:00:58.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:01:03.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:03.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:03.774 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:03.841 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:03.842 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:03.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:03.980+0000 7f69d2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:03.982 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:03.983+0000 7f69d2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:04.042 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:03Z"} 2024-03-04T19:01:04.042 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:03Z"} 2024-03-04T19:01:04.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:04.043 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:04.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:04.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:01:09.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:09.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:09.045 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:09.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:09.109 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:09.168 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:09.169+0000 7f7f97f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:09.169 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:09.169+0000 7f7f97f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:09.194 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:09Z"} 2024-03-04T19:01:09.194 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:09Z"} 2024-03-04T19:01:09.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:09.194 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:09.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:09.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:01:14.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:14.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:14.196 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:14.244 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:14.244 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:14.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:14.296+0000 7f54ec12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:14.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:14.297+0000 7f54ec12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:14.319 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:14Z"} 2024-03-04T19:01:14.319 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:14Z"} 2024-03-04T19:01:14.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:14.320 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:14.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:14.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T19:01:19.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:19.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:19.321 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:19.368 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:19.368 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:19.419+0000 7f74d9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:19.420+0000 7f74d9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:19.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:19Z"} 2024-03-04T19:01:19.443 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:19Z"} 2024-03-04T19:01:19.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:19.443 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:19.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:19.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:01:24.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:24.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:24.445 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:24.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:24.550+0000 7f2256731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:24.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:24.551+0000 7f2256731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:24.576 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:24Z"} 2024-03-04T19:01:24.576 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:24Z"} 2024-03-04T19:01:24.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:24.577 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:24.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:24.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T19:01:29.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:29.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:29.578 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:29.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:29.675+0000 7f7e7e89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:29.676+0000 7f7e7e89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:29Z"} 2024-03-04T19:01:29.699 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:29Z"} 2024-03-04T19:01:29.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:29.699 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:29.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:29.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:01:34.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:34.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:34.701 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:34.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:34.748 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:34.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:34.802+0000 7f5ab0a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:34.803 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:34.803+0000 7f5ab0a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:34.840 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:34Z"} 2024-03-04T19:01:34.840 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:34Z"} 2024-03-04T19:01:34.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:34.841 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:34.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:34.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:01:39.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:39.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:39.843 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:39.942+0000 7f6249f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:39.943+0000 7f6249f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:39.968 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:39Z"} 2024-03-04T19:01:39.969 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:39Z"} 2024-03-04T19:01:39.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:39.969 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:39.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:39.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T19:01:44.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:44.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:44.970 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:45.017 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:45.017 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:45.065 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:45.065+0000 7f64d7730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:45.066 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:45.066+0000 7f64d7730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:45.089 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:45Z"} 2024-03-04T19:01:45.089 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:45Z"} 2024-03-04T19:01:45.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:45.090 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:45.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:45.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:01:50.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:50.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:50.091 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:50.138 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:50.139 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:50.197 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:50.197+0000 7f451402d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:50.198 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:50.198+0000 7f451402d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:50.220 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:50Z"} 2024-03-04T19:01:50.220 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:50Z"} 2024-03-04T19:01:50.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:50.221 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:50.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:50.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:01:55.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:55.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:01:55.223 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:01:55.271 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:01:55.271 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:01:55.329 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:55.330+0000 7f8c0042e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:01:55.331+0000 7f8c0042e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:01:55.354 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:55Z"} 2024-03-04T19:01:55.355 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:55Z"} 2024-03-04T19:01:55.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:01:55.355 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:55.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:01:55.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.154969_713.1 2024-03-04T19:02:00.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:00.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:00.357 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:00.404 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:00.404 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:00.459 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:00.459+0000 7f13c109d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:00.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:00.460+0000 7f13c109d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:00.482 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:00Z"} 2024-03-04T19:02:00.483 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:00Z"} 2024-03-04T19:02:00.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:00.483 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:00.483 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:00.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:02:05.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:05.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:05.485 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:05.595 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:05.595+0000 7f48e360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:05.596 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:05.596+0000 7f48e360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:05.620 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:05Z"} 2024-03-04T19:02:05.621 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:05Z"} 2024-03-04T19:02:05.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:05.621 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:05.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:05.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:02:10.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:10.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:10.623 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:10.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:10.675 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:10.731+0000 7f75fb90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:10.732+0000 7f75fb90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:10.802 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:10Z"} 2024-03-04T19:02:10.802 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:10Z"} 2024-03-04T19:02:10.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:10.802 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:10.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:10.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:02:15.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:15.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:15.804 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:15.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:15.877 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:16.030 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:16.030+0000 7f23e7c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:16.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:16.031+0000 7f23e7c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:16.079 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:16Z"} 2024-03-04T19:02:16.079 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:16Z"} 2024-03-04T19:02:16.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:16.079 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:16.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:16.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:02:21.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:21.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:21.082 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:21.127 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:21.128 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:21.187 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:21.187+0000 7fdfaf2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:21.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:21.188+0000 7fdfaf2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:21.221 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:21Z"} 2024-03-04T19:02:21.222 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:21Z"} 2024-03-04T19:02:21.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:21.222 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:21.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:21.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:02:26.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:26.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:26.224 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:26.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:26.270 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:26.323+0000 7f7cf4814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:26.323 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:26.324+0000 7f7cf4814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:26.355 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:26Z"} 2024-03-04T19:02:26.356 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:26Z"} 2024-03-04T19:02:26.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:26.356 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:26.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:26.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:02:31.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:31.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:31.358 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:31.405 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:31.405 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:31.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:31.463+0000 7ff147533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:31.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:31.464+0000 7ff147533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:31.487 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:31Z"} 2024-03-04T19:02:31.487 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:31Z"} 2024-03-04T19:02:31.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:31.487 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:31.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:31.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:02:36.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:36.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:36.489 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:36.536 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:36.537 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:36.601 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:36.602+0000 7f3fad236800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:36.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:36.602+0000 7f3fad236800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:36.626 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:36Z"} 2024-03-04T19:02:36.626 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:36Z"} 2024-03-04T19:02:36.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:36.627 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:36.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:36.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:02:41.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:41.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:41.628 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:41.677 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:41.677 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:41.740 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:41.740+0000 7f7ba4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:41.740 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:41.741+0000 7f7ba4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:41.765 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:41Z"} 2024-03-04T19:02:41.765 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:41Z"} 2024-03-04T19:02:41.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:41.766 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:41.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:41.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:02:46.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:46.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:46.768 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:46.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:46.886+0000 7fa1a0244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:46.886 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:46.887+0000 7fa1a0244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:46.916 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:46Z"} 2024-03-04T19:02:46.917 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:46Z"} 2024-03-04T19:02:46.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:46.917 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:46.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:46.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:02:51.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:51.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:51.919 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:51.967 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:51.967 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:52.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:52.028+0000 7fcc94b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:52.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:52.029+0000 7fcc94b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:52.060 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:52Z"} 2024-03-04T19:02:52.060 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:52Z"} 2024-03-04T19:02:52.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:52.060 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:52.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:52.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:02:57.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:57.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:02:57.063 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:02:57.114 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:02:57.114 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:02:57.164 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:57.164+0000 7f813d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:57.165 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:02:57.165+0000 7f813d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:02:57.188 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:57Z"} 2024-03-04T19:02:57.189 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:57Z"} 2024-03-04T19:02:57.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:02:57.189 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:57.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:02:57.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:03:02.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:02.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:02.191 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:02.298 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:02.298+0000 7f33e3a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:02.299 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:02.299+0000 7f33e3a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:02.323 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:02Z"} 2024-03-04T19:03:02.324 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:02Z"} 2024-03-04T19:03:02.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:02.324 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:02.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:02.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:03:07.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:07.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:07.325 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:07.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:07.374 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:07.439+0000 7fd51f937800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:07.439 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:07.440+0000 7fd51f937800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:07.463 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:07Z"} 2024-03-04T19:03:07.464 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:07Z"} 2024-03-04T19:03:07.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:07.464 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:07.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:07.464 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:03:12.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:12.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:12.465 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:12.534 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:12.534 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:12.645 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:12.646+0000 7fcb24c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:12.646 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:12.646+0000 7fcb24c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:12Z"} 2024-03-04T19:03:12.706 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:12Z"} 2024-03-04T19:03:12.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:12.707 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:12.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:12.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:03:17.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:17.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:17.708 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:17.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:17.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:17.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:17.819+0000 7f3979a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:17.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:17.820+0000 7f3979a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:17.843 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:17Z"} 2024-03-04T19:03:17.844 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:17Z"} 2024-03-04T19:03:17.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:17.844 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:17.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:17.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:03:22.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:22.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:22.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:22.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:22.895 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:22.947 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:22.947+0000 7fcc4acc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:22.948 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:22.948+0000 7fcc4acc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:22.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:22Z"} 2024-03-04T19:03:22.970 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:22Z"} 2024-03-04T19:03:22.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:22.971 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:22.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:22.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:03:27.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:27.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:27.972 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:28.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:28.019 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:28.081 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:28.082+0000 7fbdf614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:28.082 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:28.083+0000 7fbdf614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:28.109 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:28Z"} 2024-03-04T19:03:28.109 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:28Z"} 2024-03-04T19:03:28.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:28.109 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:28.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:28.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:03:33.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:33.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:33.111 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:33.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:33.227+0000 7f2dcec12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:33.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:33.228+0000 7f2dcec12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:33.252 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:33Z"} 2024-03-04T19:03:33.252 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:33Z"} 2024-03-04T19:03:33.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:33.253 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:33.253 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:33.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:03:38.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:38.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:38.255 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:38.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:38.364+0000 7f6f00824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:38.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:38.365+0000 7f6f00824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:38.389 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:38Z"} 2024-03-04T19:03:38.389 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:38Z"} 2024-03-04T19:03:38.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:38.389 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:38.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:38.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:03:43.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:43.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:43.391 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:43.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:43.444 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:43.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:43.509+0000 7f518b11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:43.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:43.510+0000 7f518b11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:43.534 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:43Z"} 2024-03-04T19:03:43.534 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:43Z"} 2024-03-04T19:03:43.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:43.534 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:43.534 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:43.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:03:48.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:48.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:48.537 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:48.590 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:48.590 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:48.654+0000 7f7627814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:48.655 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:48.655+0000 7f7627814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:48.680 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:48Z"} 2024-03-04T19:03:48.680 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:48Z"} 2024-03-04T19:03:48.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:48.680 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:48.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:48.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:03:53.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:53.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:53.682 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:53.807 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:53.807+0000 7f32fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:53.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:53.808+0000 7f32fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:53.839 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:53Z"} 2024-03-04T19:03:53.839 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:53Z"} 2024-03-04T19:03:53.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:53.839 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:53.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:53.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:03:58.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:58.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:03:58.841 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:03:58.894 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:03:58.894 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:03:58.948 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:58.948+0000 7fc32bf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:58.948 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:03:58.949+0000 7fc32bf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:03:59.010 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:58Z"} 2024-03-04T19:03:59.011 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:58Z"} 2024-03-04T19:03:59.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:03:59.011 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:59.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:03:59.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:04:04.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:04.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:04.012 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:04.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:04.101 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:04.226 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:04.227+0000 7f86d651e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:04.227+0000 7f86d651e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:04.254 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:04Z"} 2024-03-04T19:04:04.255 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:04Z"} 2024-03-04T19:04:04.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:04.255 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:04.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:04.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T19:04:09.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:09.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:09.258 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:09.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:09.304 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:09.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:09.353+0000 7f05d5034800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:09.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:09.353+0000 7f05d5034800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:09.386 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:09Z"} 2024-03-04T19:04:09.387 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:09Z"} 2024-03-04T19:04:09.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:09.387 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:09.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:09.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:04:14.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:14.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:14.390 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:14.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:14.436 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:14.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:14.490+0000 7fee00430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:14.491 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:14.491+0000 7fee00430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:14Z"} 2024-03-04T19:04:14.515 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:14Z"} 2024-03-04T19:04:14.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:14.515 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:14.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:14.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:04:19.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:19.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:19.517 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:19.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:19.612+0000 7f547ec9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:19.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:19.613+0000 7f547ec9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:19.636 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:19Z"} 2024-03-04T19:04:19.637 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:19Z"} 2024-03-04T19:04:19.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:19.637 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:19.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:19.637 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:04:24.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:24.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:24.639 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:24.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:24.686 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:24.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:24.740+0000 7fb8d7811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:24.740 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:24.740+0000 7fb8d7811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:24.763 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:24Z"} 2024-03-04T19:04:24.763 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:24Z"} 2024-03-04T19:04:24.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:24.764 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:24.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:24.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:04:29.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:29.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:29.766 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:29.817 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:29.818 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:29.879 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:29.879+0000 7f056a230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:29.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:29.880+0000 7f056a230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:29.912 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:29Z"} 2024-03-04T19:04:29.912 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:29Z"} 2024-03-04T19:04:29.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:29.913 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:29.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:29.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:04:34.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:34.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:34.915 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:34.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:34.963 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:35.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:35.012+0000 7f72e4b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:35.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:35.013+0000 7f72e4b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:35.038 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:35Z"} 2024-03-04T19:04:35.038 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:35Z"} 2024-03-04T19:04:35.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:35.038 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:35.039 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:35.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:04:40.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:40.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:40.040 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:40.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:40.089 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:40.144 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:40.145+0000 7f5f6002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:40.145 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:40.145+0000 7f5f6002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:40.170 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:40Z"} 2024-03-04T19:04:40.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:40Z"} 2024-03-04T19:04:40.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:40.171 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:40.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:40.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:04:45.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:45.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:45.175 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:45.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:45.222 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:45.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:45.281+0000 7ffbf0f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:45.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:45.282+0000 7ffbf0f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:45.304 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:45Z"} 2024-03-04T19:04:45.305 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:45Z"} 2024-03-04T19:04:45.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:45.305 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:45.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:45.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:04:50.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:50.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:50.307 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:50.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:50.357 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:50.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:50.417+0000 7f4fa0f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:50.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:50.418+0000 7f4fa0f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:50.444 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:50Z"} 2024-03-04T19:04:50.444 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:50Z"} 2024-03-04T19:04:50.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:50.445 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:50.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:50.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:04:55.448 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-03-04T19:04:55.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:55.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:55.449 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:04:55.667 INFO:teuthology.orchestra.run.smithi027.stdout:[{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:04:55.667 DEBUG:rgw_multi.tests:master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:55.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-03-04T19:04:55.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:55.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:04:55.668 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:04:55.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:04:55.720 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:04:55.782 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:55.782+0000 7fccd2509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:55.783 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:04:55.783+0000 7fccd2509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:04:55.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:55Z"} 2024-03-04T19:04:55.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:55Z"} 2024-03-04T19:04:55.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:04:55.809 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:55.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:04:55.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:05:00.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:00.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:00.811 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:00.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:00.891 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:01.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:01.003+0000 7f0ef5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:01.004 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:01.004+0000 7f0ef5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:01.030 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:01Z"} 2024-03-04T19:05:01.030 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:01Z"} 2024-03-04T19:05:01.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:01.031 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:01.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:01.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:05:06.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:06.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:06.033 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:06.080 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:06.080 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:06.134 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:06.134+0000 7f4fedc17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:06.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:06.135+0000 7f4fedc17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:06.160 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:06Z"} 2024-03-04T19:05:06.160 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:06Z"} 2024-03-04T19:05:06.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:06.161 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:06.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:06.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:05:11.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:11.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:11.162 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:11.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:11.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:11.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:11.254+0000 7f92f8130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:11.255 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:11.255+0000 7f92f8130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:11.279 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:11Z"} 2024-03-04T19:05:11.280 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:11Z"} 2024-03-04T19:05:11.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:11.280 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:11.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:11.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:05:16.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:16.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:16.282 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:16.386 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:16.387+0000 7f5b02613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:16.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:16.388+0000 7f5b02613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:16.410 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:16Z"} 2024-03-04T19:05:16.411 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:16Z"} 2024-03-04T19:05:16.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:16.411 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:16.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:16.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:05:21.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:21.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:21.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:21.509 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:21.510+0000 7f7471933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:21.510 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:21.511+0000 7f7471933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:21.533 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:21Z"} 2024-03-04T19:05:21.533 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:21Z"} 2024-03-04T19:05:21.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:21.533 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:21.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:21.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:05:26.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:26.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:26.535 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:26.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:26.582 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:26.636 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:26.637+0000 7f2121239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:26.637 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:26.638+0000 7f2121239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:26.661 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:26Z"} 2024-03-04T19:05:26.661 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:26Z"} 2024-03-04T19:05:26.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:26.661 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:26.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:26.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:05:31.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:31.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:31.662 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:31.708 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:31.709 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:31.780 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:31.780+0000 7ff569424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:31.781 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:31.781+0000 7ff569424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:31.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:31Z"} 2024-03-04T19:05:31.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:31Z"} 2024-03-04T19:05:31.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:31.810 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:31.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:31.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:05:36.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:36.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:36.812 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:36.860 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:36.861 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:36.918 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:36.918+0000 7fb4e8467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:36.919 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:36.919+0000 7fb4e8467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:36.944 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:36Z"} 2024-03-04T19:05:36.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:36Z"} 2024-03-04T19:05:36.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:36.945 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:36.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:36.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:05:41.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:41.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:41.946 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:41.994 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:41.995 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:42.046+0000 7f9bd9813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:42.047+0000 7f9bd9813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:42.070 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:42Z"} 2024-03-04T19:05:42.070 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:42Z"} 2024-03-04T19:05:42.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:42.070 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:42.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:42.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:05:47.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:47.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:47.072 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:47.120 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:47.120 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:47.179+0000 7f7389283800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:47.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:47.180+0000 7f7389283800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:47.205 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:47Z"} 2024-03-04T19:05:47.205 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:47Z"} 2024-03-04T19:05:47.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:47.206 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:47.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:47.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:05:52.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:52.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:52.208 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:52.261 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:52.261 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:52.331 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:52.333+0000 7fccff42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:52.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:52.334+0000 7fccff42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:52.357 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:52Z"} 2024-03-04T19:05:52.358 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:52Z"} 2024-03-04T19:05:52.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:52.358 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:52.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:52.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:05:57.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:57.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:05:57.360 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:05:57.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:05:57.413 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:05:57.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:57.477+0000 7f5dcd816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:57.477 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:05:57.478+0000 7f5dcd816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:05:57.506 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:57Z"} 2024-03-04T19:05:57.506 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:57Z"} 2024-03-04T19:05:57.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:05:57.507 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:57.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:05:57.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:06:02.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:02.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:02.509 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:02.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:02.575 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:02.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:02.723+0000 7f29da133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:02.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:02.724+0000 7f29da133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:02.762 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:02Z"} 2024-03-04T19:06:02.763 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:02Z"} 2024-03-04T19:06:02.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:02.763 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:02.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:02.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:06:07.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:07.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:07.764 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:07.810 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:07.811 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:07.874+0000 7f5a611a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:07.875+0000 7f5a611a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:07.904 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:07Z"} 2024-03-04T19:06:07.904 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:07Z"} 2024-03-04T19:06:07.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:07.905 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:07.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:07.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:06:12.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:12.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:12.907 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:12.959 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:12.959 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:13.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:13.021+0000 7f221c72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:13.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:13.022+0000 7f221c72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:13.045 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:13Z"} 2024-03-04T19:06:13.045 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:13Z"} 2024-03-04T19:06:13.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.159615_716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.159615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:13.045 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:13.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.159615_716.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:13.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.159615_716.1 2024-03-04T19:06:18.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:18.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:18.047 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:18.094 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:18.094 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:18.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:18.156+0000 7f9c7f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:18.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:18.156+0000 7f9c7f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:18.188 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:18Z"} 2024-03-04T19:06:18.188 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:18Z"} 2024-03-04T19:06:18.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:18.189 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:18.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:18.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:06:23.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:23.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:23.190 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:23.237 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:23.239 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:23.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:23.304+0000 7f1927244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:23.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:23.305+0000 7f1927244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:23.328 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:23Z"} 2024-03-04T19:06:23.329 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:23Z"} 2024-03-04T19:06:23.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:23.329 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:23.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:23.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:06:28.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:28.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:28.331 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:28.379 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:28.380 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:28.432+0000 7f9757e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:28.433 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:28.433+0000 7f9757e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:28.456 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:28Z"} 2024-03-04T19:06:28.457 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:28Z"} 2024-03-04T19:06:28.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:28.457 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:28.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:28.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:06:33.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:33.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:33.459 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:33.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:33.511 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:33.579 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:33.579+0000 7f824bc38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:33.580 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:33.580+0000 7f824bc38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:33.603 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:33Z"} 2024-03-04T19:06:33.604 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:33Z"} 2024-03-04T19:06:33.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:33.604 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:33.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:33.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T19:06:38.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:38.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:38.605 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:38.656 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:38.656 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:38.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:38.711+0000 7f666792b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:38.711 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:38.712+0000 7f666792b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:38.748 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:38Z"} 2024-03-04T19:06:38.748 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:38Z"} 2024-03-04T19:06:38.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:38.748 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:38.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:38.749 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:06:43.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:43.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:43.750 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:43.803 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:43.803 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:43.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:43.865+0000 7fa5d5d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:43.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:43.866+0000 7fa5d5d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:43.900 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:43Z"} 2024-03-04T19:06:43.901 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:43Z"} 2024-03-04T19:06:43.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:43.901 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:43.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:43.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T19:06:48.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:48.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:48.903 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:49.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:49.019+0000 7fb27d022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:49.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:49.020+0000 7fb27d022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:49.128 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:49Z"} 2024-03-04T19:06:49.128 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:49Z"} 2024-03-04T19:06:49.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:49.129 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:49.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:49.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:06:54.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:54.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:54.131 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:54.208 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:54.209 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:54.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:54.316+0000 7f18f3220800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:54.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:54.316+0000 7f18f3220800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:54.369 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:54Z"} 2024-03-04T19:06:54.370 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:54Z"} 2024-03-04T19:06:54.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:54.370 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:54.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:54.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:06:59.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:59.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:06:59.372 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:06:59.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:06:59.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:06:59.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:59.492+0000 7fd08ee8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:59.493 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:06:59.493+0000 7fd08ee8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:06:59.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:59Z"} 2024-03-04T19:06:59.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:59Z"} 2024-03-04T19:06:59.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:06:59.530 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:59.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:06:59.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:07:04.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:04.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:04.532 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:04.640 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:04.640+0000 7fe44b012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:04.641 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:04.641+0000 7fe44b012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:04.667 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:04Z"} 2024-03-04T19:07:04.667 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:04Z"} 2024-03-04T19:07:04.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:07:04.667 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:04.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:04.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:07:09.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:09.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:09.669 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:09.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:09.716 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:09.767 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:09.767+0000 7fd37cb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:09.768+0000 7fd37cb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:09.792 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:09Z"} 2024-03-04T19:07:09.792 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:09Z"} 2024-03-04T19:07:09.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:07:09.792 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:09.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:09.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T19:07:14.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:14.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:14.794 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:14.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:14.894+0000 7f4910c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:14.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:14.895+0000 7f4910c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:14.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:14Z"} 2024-03-04T19:07:14.923 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:14Z"} 2024-03-04T19:07:14.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:07:14.924 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:14.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:14.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:07:19.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:19.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:19.925 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:19.976 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:19.977 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:20.036 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:20.037+0000 7f8718f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:20.038 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:20.038+0000 7f8718f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:20.065 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:20Z"} 2024-03-04T19:07:20.065 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:20Z"} 2024-03-04T19:07:20.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:07:20.065 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:20.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:20.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:07:25.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:25.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:25.067 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:07:25.116 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-03-04T19:07:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-03-04T19:07:25.172 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:25.173+0000 7fefe8c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:25.173 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-04T19:07:25.174+0000 7fefe8c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:07:25.198 INFO:teuthology.orchestra.run.smithi071.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:25Z"} 2024-03-04T19:07:25.198 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:25Z"} 2024-03-04T19:07:25.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:07:25.198 DEBUG:rgw_multi.tests:log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:25.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:07:25.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:07:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~0s 2024-03-04T19:07:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2024-03-04T19:07:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~0s 2024-03-04T19:07:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~0s 2024-03-04T19:07:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T19:07:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-03-04T19:07:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T19:07:28.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~0s 2024-03-04T19:07:30.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:30.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:07:30.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:07:30.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:07:30.205 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:07:32.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~5s 2024-03-04T19:07:32.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~5s 2024-03-04T19:07:33.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~5s 2024-03-04T19:07:33.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~5s 2024-03-04T19:07:33.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~5s 2024-03-04T19:07:33.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~5s 2024-03-04T19:07:33.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~5s 2024-03-04T19:07:33.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~5s 2024-03-04T19:07:33.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~5s 2024-03-04T19:07:37.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~10s 2024-03-04T19:07:37.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~10s 2024-03-04T19:07:37.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~10s 2024-03-04T19:07:37.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~10s 2024-03-04T19:07:37.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~10s 2024-03-04T19:07:37.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~10s 2024-03-04T19:07:37.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~10s 2024-03-04T19:07:38.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~10s 2024-03-04T19:07:39.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:07:39.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~15s 2024-03-04T19:07:42.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~15s 2024-03-04T19:07:43.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~15s 2024-03-04T19:07:43.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~15s 2024-03-04T19:07:47.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~20s 2024-03-04T19:07:47.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~20s 2024-03-04T19:07:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~20s 2024-03-04T19:07:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~20s 2024-03-04T19:07:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~20s 2024-03-04T19:07:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~20s 2024-03-04T19:07:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~20s 2024-03-04T19:07:48.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~20s 2024-03-04T19:07:48.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~20s 2024-03-04T19:07:52.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~25s 2024-03-04T19:07:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~25s 2024-03-04T19:07:53.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~25s 2024-03-04T19:07:53.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~25s 2024-03-04T19:07:57.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~30s 2024-03-04T19:07:57.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~30s 2024-03-04T19:07:57.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~30s 2024-03-04T19:07:57.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~30s 2024-03-04T19:07:57.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~30s 2024-03-04T19:07:57.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~30s 2024-03-04T19:07:57.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~30s 2024-03-04T19:07:57.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:07:58.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~30s 2024-03-04T19:07:58.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~30s 2024-03-04T19:07:58.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~30s 2024-03-04T19:07:58.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~30s 2024-03-04T19:07:58.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~30s 2024-03-04T19:07:58.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~30s 2024-03-04T19:07:58.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~30s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~35s 2024-03-04T19:08:02.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~35s 2024-03-04T19:08:03.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~35s 2024-03-04T19:08:03.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~35s 2024-03-04T19:08:07.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~40s 2024-03-04T19:08:07.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~40s 2024-03-04T19:08:07.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~40s 2024-03-04T19:08:07.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~40s 2024-03-04T19:08:07.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~40s 2024-03-04T19:08:07.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~40s 2024-03-04T19:08:07.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~40s 2024-03-04T19:08:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~40s 2024-03-04T19:08:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~40s 2024-03-04T19:08:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~40s 2024-03-04T19:08:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~40s 2024-03-04T19:08:08.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~40s 2024-03-04T19:08:08.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~40s 2024-03-04T19:08:08.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~40s 2024-03-04T19:08:09.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:08:09.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:12.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~45s 2024-03-04T19:08:12.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~45s 2024-03-04T19:08:12.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~45s 2024-03-04T19:08:12.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~45s 2024-03-04T19:08:12.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2024-03-04T19:08:12.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2024-03-04T19:08:12.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~45s 2024-03-04T19:08:12.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~45s 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~45s 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~45s 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~45s 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2024-03-04T19:08:13.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2024-03-04T19:08:13.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~45s 2024-03-04T19:08:15.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:08:15.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:15.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:08:15.772 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:08:15.774 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ERROR 2024-03-04T19:08:15.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:15.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:15.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:08:15.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2024-03-04T19:08:17.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~50s 2024-03-04T19:08:18.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~50s 2024-03-04T19:08:18.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~50s 2024-03-04T19:08:18.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~50s 2024-03-04T19:08:18.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~50s 2024-03-04T19:08:18.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2024-03-04T19:08:18.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2024-03-04T19:08:18.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~50s 2024-03-04T19:08:18.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:08:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~55s 2024-03-04T19:08:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~55s 2024-03-04T19:08:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~55s 2024-03-04T19:08:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~55s 2024-03-04T19:08:22.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~55s 2024-03-04T19:08:22.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~55s 2024-03-04T19:08:22.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~55s 2024-03-04T19:08:23.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~55s 2024-03-04T19:08:27.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~60s 2024-03-04T19:08:27.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~60s 2024-03-04T19:08:27.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:08:27.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~60s 2024-03-04T19:08:28.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~60s 2024-03-04T19:08:30.911 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:08:32.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~65s 2024-03-04T19:08:32.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~65s 2024-03-04T19:08:32.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~65s 2024-03-04T19:08:32.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~65s 2024-03-04T19:08:32.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~65s 2024-03-04T19:08:32.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~65s 2024-03-04T19:08:32.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~65s 2024-03-04T19:08:33.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~65s 2024-03-04T19:08:33.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~65s 2024-03-04T19:08:33.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~65s 2024-03-04T19:08:33.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~65s 2024-03-04T19:08:33.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~65s 2024-03-04T19:08:33.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~65s 2024-03-04T19:08:33.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~65s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~70s 2024-03-04T19:08:37.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~70s 2024-03-04T19:08:38.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~70s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~75s 2024-03-04T19:08:42.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~75s 2024-03-04T19:08:42.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:08:42.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:43.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~75s 2024-03-04T19:08:43.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~75s 2024-03-04T19:08:43.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~75s 2024-03-04T19:08:43.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~75s 2024-03-04T19:08:43.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~75s 2024-03-04T19:08:43.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~75s 2024-03-04T19:08:43.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~75s 2024-03-04T19:08:45.984 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:08:47.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~80s 2024-03-04T19:08:47.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~80s 2024-03-04T19:08:47.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~80s 2024-03-04T19:08:47.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~80s 2024-03-04T19:08:47.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~80s 2024-03-04T19:08:47.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~80s 2024-03-04T19:08:47.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~80s 2024-03-04T19:08:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~80s 2024-03-04T19:08:48.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:08:48.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:49.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:08:49.052 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:08:49.053 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ERROR 2024-03-04T19:08:49.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:49.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:08:49.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:08:49.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:08:52.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:08:52.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~85s 2024-03-04T19:08:52.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~85s 2024-03-04T19:08:52.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~85s 2024-03-04T19:08:52.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~85s 2024-03-04T19:08:52.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~85s 2024-03-04T19:08:52.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~85s 2024-03-04T19:08:52.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~85s 2024-03-04T19:08:53.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~85s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2024-03-04T19:08:57.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2024-03-04T19:08:58.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~90s 2024-03-04T19:09:01.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:01.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:02.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~95s 2024-03-04T19:09:02.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~95s 2024-03-04T19:09:02.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~95s 2024-03-04T19:09:02.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~95s 2024-03-04T19:09:02.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2024-03-04T19:09:02.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~95s 2024-03-04T19:09:02.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2024-03-04T19:09:03.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~95s 2024-03-04T19:09:04.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~100s 2024-03-04T19:09:07.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~100s 2024-03-04T19:09:08.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~100s 2024-03-04T19:09:12.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~105s 2024-03-04T19:09:12.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~105s 2024-03-04T19:09:12.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~105s 2024-03-04T19:09:12.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~105s 2024-03-04T19:09:12.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~105s 2024-03-04T19:09:12.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~105s 2024-03-04T19:09:12.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~105s 2024-03-04T19:09:13.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~105s 2024-03-04T19:09:16.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:16.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~110s 2024-03-04T19:09:17.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~110s 2024-03-04T19:09:18.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~110s 2024-03-04T19:09:18.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~110s 2024-03-04T19:09:19.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:22.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:22.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~115s 2024-03-04T19:09:22.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~115s 2024-03-04T19:09:22.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:22.332 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:22.333 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ERROR 2024-03-04T19:09:22.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:22.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:22.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:22.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~115s 2024-03-04T19:09:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~115s 2024-03-04T19:09:25.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:27.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~120s 2024-03-04T19:09:27.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~120s 2024-03-04T19:09:28.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~120s 2024-03-04T19:09:28.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~120s 2024-03-04T19:09:28.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~120s 2024-03-04T19:09:28.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~120s 2024-03-04T19:09:28.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~120s 2024-03-04T19:09:28.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~120s 2024-03-04T19:09:28.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~120s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~125s 2024-03-04T19:09:32.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~125s 2024-03-04T19:09:33.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~125s 2024-03-04T19:09:33.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~125s 2024-03-04T19:09:34.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:34.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:37.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~130s 2024-03-04T19:09:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~130s 2024-03-04T19:09:37.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:38.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~130s 2024-03-04T19:09:38.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~130s 2024-03-04T19:09:42.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~135s 2024-03-04T19:09:42.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~135s 2024-03-04T19:09:42.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~135s 2024-03-04T19:09:42.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~135s 2024-03-04T19:09:42.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2024-03-04T19:09:42.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~135s 2024-03-04T19:09:42.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2024-03-04T19:09:43.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~135s 2024-03-04T19:09:47.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~140s 2024-03-04T19:09:47.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~140s 2024-03-04T19:09:48.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~140s 2024-03-04T19:09:48.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~140s 2024-03-04T19:09:48.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~140s 2024-03-04T19:09:48.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~140s 2024-03-04T19:09:48.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2024-03-04T19:09:48.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2024-03-04T19:09:48.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~140s 2024-03-04T19:09:49.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:49.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~145s 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~145s 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~145s 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~145s 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~145s 2024-03-04T19:09:52.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~145s 2024-03-04T19:09:52.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~145s 2024-03-04T19:09:52.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~145s 2024-03-04T19:09:53.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~145s 2024-03-04T19:09:55.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.615 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:55.615 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ERROR 2024-03-04T19:09:55.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create_rgw_down ... SKIP: test_bucket_create_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove_rgw_down ... SKIP: test_bucket_remove_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync_rgw_down ... SKIP: test_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete_rgw_down ... SKIP: test_object_delete_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync_rgw_down ... SKIP: test_concurrent_versioned_object_incremental_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync_rgw_down ... SKIP: test_suspended_delete_marker_full_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl_rgw_down ... SKIP: test_bucket_acl_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable_rgw_down ... SKIP: test_bucket_sync_enable_right_after_disable_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync_rgw_down ... SKIP: test_multipart_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_run_basic_incremental_rgw_down ... SKIP: test_bucket_sync_run_basic_incremental_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync_rgw_down ... SKIP: test_role_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init_rgw_down ... SKIP: test_bucket_full_sync_after_data_sync_init_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_policy_config_zonegroup_rgw_down ... SKIP: test_sync_policy_config_zonegroup_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_flow_symmetrical_zonegroup_all_rgw_down ... SKIP: test_sync_flow_symmetrical_zonegroup_all_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_single_bucket: 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:55.623 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2442, in test_sync_single_bucket 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 185, in zone_meta_checkpoint 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 139, in meta_sync_status 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: meta_sync_status_json, retcode = zone.cluster.admin(cmd, check_retcode=False, read_only=True) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:09:55.624 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi071 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:55.782+0000 7fccd2509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:55.783+0000 7fccd2509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:55Z"} 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:55Z"} 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:01.003+0000 7f0ef5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:01.004+0000 7f0ef5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:01Z"} 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:01Z"} 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:06.134+0000 7f4fedc17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:06.135+0000 7f4fedc17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:06Z"} 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:06Z"} 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:11.254+0000 7f92f8130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:11.255+0000 7f92f8130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:11Z"} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:11Z"} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:16.387+0000 7f5b02613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:16.388+0000 7f5b02613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:16Z"} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:16Z"} 2024-03-04T19:09:55.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:21.510+0000 7f7471933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:21.511+0000 7f7471933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:21Z"} 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:21Z"} 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:26.637+0000 7f2121239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:26.638+0000 7f2121239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:26Z"} 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:26Z"} 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:55.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:31.780+0000 7ff569424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:31.781+0000 7ff569424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:31Z"} 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:31Z"} 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:36.918+0000 7fb4e8467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:36.919+0000 7fb4e8467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:36Z"} 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:36Z"} 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:42.046+0000 7f9bd9813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:42.047+0000 7f9bd9813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:42Z"} 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:42Z"} 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:47.179+0000 7f7389283800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:47.180+0000 7f7389283800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:47Z"} 2024-03-04T19:09:55.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:47Z"} 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:52.333+0000 7fccff42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:52.334+0000 7fccff42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:52Z"} 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:52Z"} 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:57.477+0000 7f5dcd816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:05:57.478+0000 7f5dcd816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:57Z"} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:05:57Z"} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:02.723+0000 7f29da133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:02.724+0000 7f29da133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:02Z"} 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:02Z"} 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:07.874+0000 7f5a611a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:07.875+0000 7f5a611a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:07Z"} 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:07Z"} 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:13.021+0000 7f221c72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:13.022+0000 7f221c72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:13Z"} 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:13Z"} 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.159615_716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.159615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.159615_716.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.159615_716.1 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:18.156+0000 7f9c7f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:18.156+0000 7f9c7f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:18Z"} 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:18Z"} 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:23.304+0000 7f1927244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:23.305+0000 7f1927244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:23Z"} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:23Z"} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:28.432+0000 7f9757e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:28.433+0000 7f9757e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:28Z"} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:28Z"} 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:33.579+0000 7f824bc38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:33.580+0000 7f824bc38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:33Z"} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:33Z"} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:55.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:38.711+0000 7f666792b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:38.712+0000 7f666792b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:38Z"} 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:38Z"} 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:43.865+0000 7fa5d5d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:43.866+0000 7fa5d5d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:43Z"} 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:43Z"} 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:49.019+0000 7fb27d022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:49.020+0000 7fb27d022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:49Z"} 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:49Z"} 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:54.316+0000 7f18f3220800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:54.316+0000 7f18f3220800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:54Z"} 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:54Z"} 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:59.492+0000 7fd08ee8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:06:59.493+0000 7fd08ee8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:59Z"} 2024-03-04T19:09:55.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:06:59Z"} 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:04.640+0000 7fe44b012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:04.641+0000 7fe44b012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:04Z"} 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:04Z"} 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:09.767+0000 7fd37cb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:09.768+0000 7fd37cb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:09Z"} 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:09Z"} 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:14.894+0000 7f4910c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:14.895+0000 7f4910c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:14Z"} 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:14Z"} 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:20.037+0000 7f8718f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:20.038+0000 7f8718f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:20Z"} 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:20Z"} 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:25.173+0000 7fefe8c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:07:25.174+0000 7fefe8c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:25Z"} 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:07:25Z"} 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~0s 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~0s 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~0s 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~0s 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-03-04T19:09:55.651 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~0s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~5s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~5s 2024-03-04T19:09:55.652 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~5s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~10s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~10s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~10s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~10s 2024-03-04T19:09:55.653 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~10s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~15s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~15s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~15s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~15s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~15s 2024-03-04T19:09:55.654 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~15s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~20s 2024-03-04T19:09:55.655 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~20s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~25s 2024-03-04T19:09:55.656 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~25s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~30s 2024-03-04T19:09:55.657 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~30s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~30s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~30s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~30s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~35s 2024-03-04T19:09:55.658 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~35s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~35s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~40s 2024-03-04T19:09:55.659 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~40s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~45s 2024-03-04T19:09:55.660 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~45s 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~45s 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~45s 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_different_buckets: 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:55.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2559, in test_sync_different_buckets 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~50s 2024-03-04T19:09:55.663 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~50s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~55s 2024-03-04T19:09:55.664 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~55s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~55s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~60s 2024-03-04T19:09:55.665 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~60s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~60s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~60s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~65s 2024-03-04T19:09:55.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~65s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~65s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~70s 2024-03-04T19:09:55.667 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~75s 2024-03-04T19:09:55.668 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~75s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~80s 2024-03-04T19:09:55.669 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~80s 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~80s 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_multiple_buckets_to_single: 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:55.670 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2705, in test_sync_multiple_buckets_to_single 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~85s 2024-03-04T19:09:55.672 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~85s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~85s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~90s 2024-03-04T19:09:55.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~95s 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.674 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~100s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~105s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~105s 2024-03-04T19:09:55.675 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~105s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~110s 2024-03-04T19:09:55.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~110s 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.677 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~115s 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:ERROR: test_sync_single_bucket_to_multiple: 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:55.678 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2826, in test_sync_single_bucket_to_multiple 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 205, in zonegroup_meta_checkpoint 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 149, in meta_master_log_status 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite.py", line 195, in admin 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests:ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:55.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~115s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~120s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~120s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~120s 2024-03-04T19:09:55.680 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~120s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~125s 2024-03-04T19:09:55.681 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~125s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~130s 2024-03-04T19:09:55.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~130s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~135s 2024-03-04T19:09:55.683 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~135s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~135s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~135s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~135s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~135s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~140s 2024-03-04T19:09:55.684 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~140s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~140s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~140s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.0 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.1 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.osd.2 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mon.a is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.1 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.mgr.x is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.0 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.1 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.osd.2 is failed for ~145s 2024-03-04T19:09:55.685 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mon.a is failed for ~145s 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.0 is failed for ~145s 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~145s 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.mgr.x is failed for ~145s 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: INFO: Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.connection: DEBUG: {'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.remote: DEBUG: [Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:teuthology.contextutil: WARNING: 'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-03-04T19:09:55.686 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:55.687 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 690, in test_multi_object_delete 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 389, in zone_bucket_checkpoint 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite.py", line 197, in admin 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-16 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:0oYN+vdczNc0Y6wYJlmlLXdgkb8= 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0oYN+vdczNc0Y6wYJlmlLXdgkb8=', 'Content-Length': '0'} 2024-03-04T19:09:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e784b7ad565e5e9e-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-17 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.690 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:DsNIxYORd6nUbKtOXM/Jt9ZVC54= 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DsNIxYORd6nUbKtOXM/Jt9ZVC54=', 'Content-Length': '0'} 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000263042343bd4697f-0065e5f483-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-18 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:cWHV74iBEmx/6KMszUbjISBq0Is= 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cWHV74iBEmx/6KMszUbjISBq0Is=', 'Content-Length': '0'} 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009d265bd10e152bcc-0065e5f483-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a9de997402fd24f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj1 2024-03-04T19:09:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj1 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj1 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj1 2024-03-04T19:09:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:vGmXhnpuBX7dagK3jBkQ0DTjy6c= 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vGmXhnpuBX7dagK3jBkQ0DTjy6c='} 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000564c29e96b77f91c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.695 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d3d8ff0f478aa5-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj2 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj2 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj2 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj2 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Q8abwkRHB5cYVwnfwWWCkbDgdlg= 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q8abwkRHB5cYVwnfwWWCkbDgdlg='} 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005541c1b7cb24853c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000361a23308390b84f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj3 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj3 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj3 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.699 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj3 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:EyeD5gR2AIwTXIzvXF/c6ZMzrGQ= 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EyeD5gR2AIwTXIzvXF/c6ZMzrGQ='} 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a565684f33b34aad-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af18401105cdf8b4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj4 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj4 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj4 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj4 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2Hexti9QDaSeihC00vLgRda23I4= 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2Hexti9QDaSeihC00vLgRda23I4='} 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a5c39fd30558668-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4e4763a328de9be-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj5 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj5 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj5 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.704 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj5 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:U3smLnTNyN6Cb2QnrK6PXT77Moc= 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:U3smLnTNyN6Cb2QnrK6PXT77Moc='} 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b908342aafa712-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000362072ec4030f8ae-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj6 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj6 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj6 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj6 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:LeJ/evngH8CqvNHwvP2Bpu5S/UE= 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LeJ/evngH8CqvNHwvP2Bpu5S/UE='} 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f4aeb370206dd4c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4f9b064d5741724-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj7 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj7 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj7 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.709 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj7 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kkMl6vxLC5DnYIzgztvtJ23gx2c= 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kkMl6vxLC5DnYIzgztvtJ23gx2c='} 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000207c38b974efb517-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0b67daef713a409-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj8 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj8 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj8 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj8 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O9cCJ8lB1HiJ/bLDc2bdGePUASM= 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O9cCJ8lB1HiJ/bLDc2bdGePUASM='} 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044f102e44226cf37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3a0f8962bbf5263-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj9 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj9 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj9 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.714 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj9 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wctQePK4Nyxf8gkir+MV2IdAHUQ= 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wctQePK4Nyxf8gkir+MV2IdAHUQ='} 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3e66cb42a3e6e9-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007466c633bd23a6ac-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj10 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj10 2024-03-04T19:09:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj10 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj10 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2AXDcf4WXCrmkyOpHaYWg3ALcDs= 2024-03-04T19:09:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2AXDcf4WXCrmkyOpHaYWg3ALcDs='} 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027e74d68e579c794-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.718 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092be47adc9b1245c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj11 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj11 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj11 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj11 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:cBssGGDIO2TbJZoGFJQmTyt5DgU= 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:cBssGGDIO2TbJZoGFJQmTyt5DgU='} 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004449c6f7fe25b854-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cafc349fbbe12ef-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj12 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj12 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj12 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj12 2024-03-04T19:09:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:F1TyeAL0nPAclA+D5W3ZZCGrAJY= 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:F1TyeAL0nPAclA+D5W3ZZCGrAJY='} 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081279fc71c39548b-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.723 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8f674b914fb3363-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj13 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj13 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj13 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj13 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:tE6SfcMm5E6uarTeiw6rnhxZCns= 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tE6SfcMm5E6uarTeiw6rnhxZCns='} 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000829c0064954ddc10-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b107b6b19eba6695-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj14 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj14 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj14 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj14 2024-03-04T19:09:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:d/dRYOGokRcA1QhHXdlRXExtNXg= 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d/dRYOGokRcA1QhHXdlRXExtNXg='} 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6c4544d2be67d81-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.728 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec228b226f32a3d2-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj15 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj15 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj15 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj15 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:QPPJeUd91TIVfizHtFPMBWGT6c0= 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QPPJeUd91TIVfizHtFPMBWGT6c0='} 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd0ac211c66e705d-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000237f7aedfec1fa81-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj16 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj16 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj16 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.732 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj16 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:g1ag2/RP3i75IE4pg+6BtLtF/zM= 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:g1ag2/RP3i75IE4pg+6BtLtF/zM='} 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094c6f1574f03edf1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.733 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113708b1235b6978-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj17 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj17 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj17 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj17 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:4EcFgb2fBWUfB1vPvAXtfsEgp0g= 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4EcFgb2fBWUfB1vPvAXtfsEgp0g='} 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f06f19025a03dc0-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5e0bdc2b414bd2e-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj18 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj18 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj18 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.737 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj18 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:aHvzkTHhtgtEAfOgQyCdxvKv478= 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aHvzkTHhtgtEAfOgQyCdxvKv478='} 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7bc8eba198e4003-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.738 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c6f6b449c1b6ae4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj19 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj19 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj19 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj19 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:LBl3N0j4Vj2Jmv4TfWE/LqZbn9I= 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LBl3N0j4Vj2Jmv4TfWE/LqZbn9I='} 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010a847206d0a83a1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009333bcbf868cf063-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj20 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj20 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj20 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.742 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj20 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:mZMKfjS22hQTt26ZcVjW8HEL4t0= 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mZMKfjS22hQTt26ZcVjW8HEL4t0='} 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070023ef686e011e3-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.743 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf63e5da118e300-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj21 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj21 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj21 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj21 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:UledhRQPW5YoMLeO5XWOnx4YDrU= 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UledhRQPW5YoMLeO5XWOnx4YDrU='} 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e363b62fc134171-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b481b12b0b28fdc5-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj22 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj22 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj22 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.747 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj22 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:zBRsTZCTscqKQZc5TF+1jmNJg4k= 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zBRsTZCTscqKQZc5TF+1jmNJg4k='} 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000607fba4dc1432e20-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82be12f50ed8265-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj23 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj23 2024-03-04T19:09:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj23 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj23 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.750 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Yb+uqtaX7QuP1b7voMZEYPh1B+0= 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Yb+uqtaX7QuP1b7voMZEYPh1B+0='} 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a3c4d59b1052fa4-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.751 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd54920a813065ed-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj24 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj24 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj24 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj24 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:4CMGcXDgVcb8u6ZY5VhFgPeQcBQ= 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4CMGcXDgVcb8u6ZY5VhFgPeQcBQ='} 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009499729bfa03590f-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eca8e2f1d4f734d7-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj25 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj25 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj25 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.755 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj25 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZSVtlg79kWOyqoz1+Dt4IduVGCc= 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZSVtlg79kWOyqoz1+Dt4IduVGCc='} 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f147022014a0c6df-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f657efee3538427-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj26 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj26 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj26 2024-03-04T19:09:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj26 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Tx9y2TVBm1AvPxu9qGMiHBh/A9A= 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tx9y2TVBm1AvPxu9qGMiHBh/A9A='} 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d747d9d8ff427109-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.759 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000349634e8907efdf1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj27 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj27 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj27 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.760 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj27 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wkj3ob31I8iHLaV4tnAyCl+UWAU= 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wkj3ob31I8iHLaV4tnAyCl+UWAU='} 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e64c47559a94b4a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005083e714c561e441-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj28 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj28 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj28 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj28 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:DviHUXHJFK+0InMMzseg9BoJTgw= 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DviHUXHJFK+0InMMzseg9BoJTgw='} 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029ef902db607a3f6-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007efe3df3cf338f09-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj29 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj29 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj29 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.765 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj29 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:vYZ9BYsnkVbFycgqGgSGUYKuLyc= 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vYZ9BYsnkVbFycgqGgSGUYKuLyc='} 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e96cab60153223a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000168eb9600e1e0e89-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj30 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj30 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj30 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj30 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:dF4SBUQcoipfsOnsqPuDs4jWs4o= 2024-03-04T19:09:55.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dF4SBUQcoipfsOnsqPuDs4jWs4o='} 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d92d82f3912de424-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.769 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001394385581ac5ef2-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj31 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj31 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj31 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj31 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:0h9i11fwUX5yQirV/okfui1fOTk= 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0h9i11fwUX5yQirV/okfui1fOTk='} 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005db799fc0ae01f37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c394b0219f1f6749-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj32 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj32 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj32 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.773 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj32 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:XKbEaV5QSGkggluQ1BBT3YOQGkc= 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XKbEaV5QSGkggluQ1BBT3YOQGkc='} 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000293e261a3976ff1d-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71b15fdca192506-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj33 2024-03-04T19:09:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj33 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj33 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj33 2024-03-04T19:09:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:b5gOPVnMNOQoYJtwJzm5cjmf5P0= 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:b5gOPVnMNOQoYJtwJzm5cjmf5P0='} 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcbe987617208b24-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.777 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000533bc871a6db6147-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj34 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj34 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj34 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj34 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:pyuZZRH825KsuU9N64LSV/s+N/o= 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pyuZZRH825KsuU9N64LSV/s+N/o='} 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e55dda2753c6c1ad-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082ac3e6a69bc9a37-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj35 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj35 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj35 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.781 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj35 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:mAG3tK4d677+vOWczsFQDZ64e0M= 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mAG3tK4d677+vOWczsFQDZ64e0M='} 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8f8fe914240836c-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.782 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA= 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KPjlWSWMglWEluAAYfyxMYd74zA=', 'Content-Length': '0'} 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fb6ad8f25a002c1-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj36 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj36 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj36 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:15 GMT 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj36 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:brlBzM5dw/8ikHUiv57mRbDchv4= 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:15 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:brlBzM5dw/8ikHUiv57mRbDchv4='} 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009027de3d1241d40a-0065e5f483-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:15 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000348848636c709071-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj37 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj37 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj37 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.786 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj37 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:590epzfjK/rQUK5fYEpDyX70Uas= 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:590epzfjK/rQUK5fYEpDyX70Uas='} 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092afaae54b3d38d8-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbda926a261da02a-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj38 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj38 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj38 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj38 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:uwnuZPMneQNDNIMcZVbZL6LwoU8= 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uwnuZPMneQNDNIMcZVbZL6LwoU8='} 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cd5e5ea0a7cfbee-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.790 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c1a273c21c0f7c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj39 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj39 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj39 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.791 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj39 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:4waapvH3BJF2a6OAsFh3TxshjNk= 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4waapvH3BJF2a6OAsFh3TxshjNk='} 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dfd627d54a71e60-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008055af68f1897ce2-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj40 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj40 2024-03-04T19:09:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj40 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj40 2024-03-04T19:09:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TTPfBUt7hnbs4ViTPboZK4F0Q3M= 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TTPfBUt7hnbs4ViTPboZK4F0Q3M='} 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000992a64b8f7ce3120-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.795 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013d22651c45b5faf-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj41 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj41 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj41 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj41 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Uu41hN+Gv4Xammmp8U+7VU+Fs3k= 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uu41hN+Gv4Xammmp8U+7VU+Fs3k='} 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6843e996e4673de-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f82cb6569e40d66-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj42 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj42 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj42 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj42 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:QiPQf6sYZQh4GMN3/zDhSNdC+0M= 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QiPQf6sYZQh4GMN3/zDhSNdC+0M='} 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a84193703169a826-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038d8f9b3f68007a3-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj43 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj43 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj43 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.802 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj43 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2a4zWh3P0b4He2qqljk8VMve9Us= 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2a4zWh3P0b4He2qqljk8VMve9Us='} 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000406617fedbcb73b7-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3cf1addb4a0c7ec-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj44 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj44 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj44 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj44 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:zzBMIdwsx6oM5/1PTMvTIFeJQ88= 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zzBMIdwsx6oM5/1PTMvTIFeJQ88='} 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067205b98de3b359a-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000658f6a993f02cba4-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj45 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj45 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj45 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.807 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj45 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:t1F7wks54+26W5jEI2zAmqA6yww= 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t1F7wks54+26W5jEI2zAmqA6yww='} 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000825f2f9664dfd4a0-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b47377a53fb1d22-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj46 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj46 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj46 2024-03-04T19:09:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj46 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Ev+UtUIfghsnxv7Y/Kr/araCJZ0= 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ev+UtUIfghsnxv7Y/Kr/araCJZ0='} 2024-03-04T19:09:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acfde32f6ca7b8dd-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.811 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0697f821ac1b76e-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj47 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj47 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj47 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj47 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:zGg0ymRHkYFYaA+SQ8mbzLmoGpE= 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zGg0ymRHkYFYaA+SQ8mbzLmoGpE='} 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000519569b88a4b531c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5ae95673aeea280-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj48 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj48 2024-03-04T19:09:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj48 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj48 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.815 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5DB52MqYofpLT+pV9YsoqJCYiSo= 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5DB52MqYofpLT+pV9YsoqJCYiSo='} 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9787bb56425021c-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/ 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/ 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/ 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.816 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:/sbnvju-16/ 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8= 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZudKfl+cwyOOrXj7ot6Xcn3D8e8=', 'Content-Length': '0'} 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d1c0b0523c01422-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-16/obj49 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-16/obj49 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-16/obj49 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:/sbnvju-16/obj49 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:DGhZW4+okHkQqifvLDL64Tvb+7w= 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:DGhZW4+okHkQqifvLDL64Tvb+7w='} 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097f002373c8571ef-0065e5f484-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a9a88ff24c6633c-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj1 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj1 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj1 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.820 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj1 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:BhACC10gHBqlE8hJF2WK+rFaW04= 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BhACC10gHBqlE8hJF2WK+rFaW04='} 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000909607b39151743c-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000600180f2056cc6ac-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj2 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj2 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj2 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj2 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:4RsDKP+xZJoVr3kWloVSK0O4+3c= 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4RsDKP+xZJoVr3kWloVSK0O4+3c='} 2024-03-04T19:09:55.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042595f32e98da6bf-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003efc859bed5e36ff-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj3 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj3 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj3 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.825 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj3 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:lIF01ANDufqivajee0rp5TobYRQ= 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lIF01ANDufqivajee0rp5TobYRQ='} 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000970f76d11447b1e7-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008acf02b00e3b7fa-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj4 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj4 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj4 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj4 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Ner0x96wCW2ysbwV7YqzXvabhjc= 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ner0x96wCW2ysbwV7YqzXvabhjc='} 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf75ce4fc0a3b6c4-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.829 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bea8d9dd8458a6d-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj5 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj5 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj5 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.830 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj5 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:qFHdyqi6+wkL480pT02dS0kVVIY= 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qFHdyqi6+wkL480pT02dS0kVVIY='} 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bae1265fbbbb04a1-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ce415ef912a93d9-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj6 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj6 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj6 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj6 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:A4sn1jrI1xzt6D628Lo85pi+nCI= 2024-03-04T19:09:55.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:A4sn1jrI1xzt6D628Lo85pi+nCI='} 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ccbca55db734084-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.834 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098370d552bbf02e4-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj7 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj7 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj7 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj7 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rA9Psq8fLyEVearUvsrbsr7OUXM= 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rA9Psq8fLyEVearUvsrbsr7OUXM='} 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ebc1701022580a5-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.837 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008267478a8b78ded6-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj8 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj8 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj8 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj8 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ECaQfC1bBtQ3EskNck2TFihM+SQ= 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ECaQfC1bBtQ3EskNck2TFihM+SQ='} 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000943c644dc0cea19-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c6dbf3ce096e409-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj9 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj9 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj9 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.841 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj9 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jNKhV8edSyFZwSH4KOSKh+ACpCo= 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jNKhV8edSyFZwSH4KOSKh+ACpCo='} 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0191189fb3c15d5-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb7b7c316242084e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj10 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj10 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj10 2024-03-04T19:09:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj10 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:1+RWjymuPOmKubEkk2HSRI01ai8= 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:1+RWjymuPOmKubEkk2HSRI01ai8='} 2024-03-04T19:09:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9c742b50abea2a8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.845 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dbd72cbca835c30-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj11 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj11 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj11 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj11 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:9DkqnSUiz2s5DcyVwkKm1XKLXeQ= 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9DkqnSUiz2s5DcyVwkKm1XKLXeQ='} 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfb622ea1e86e91f-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.848 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042ae9b6c20027c36-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj12 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj12 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj12 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.849 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj12 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kq1dDRe+W0oz4FP5O6SUhKe7cZg= 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kq1dDRe+W0oz4FP5O6SUhKe7cZg='} 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cf96ce862d6d0da-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b359c176c60db81-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj13 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj13 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj13 2024-03-04T19:09:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj13 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5hdS4dSdcZXoFdH5pOEHg7kuh4E= 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5hdS4dSdcZXoFdH5pOEHg7kuh4E='} 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066d46ecb6e1bdb0b-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.853 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220afa5b75499584-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj14 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj14 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj14 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.854 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj14 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:LF3FcclFGl+OnBZT0FKkeuR/CKs= 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LF3FcclFGl+OnBZT0FKkeuR/CKs='} 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cdc09e610ac3fcb-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029b465d5de429fef-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj15 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj15 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj15 2024-03-04T19:09:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj15 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:a6OoBFADN/s7xB9o3ktkPoMvzIw= 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:a6OoBFADN/s7xB9o3ktkPoMvzIw='} 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c4de98c0d088526-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.858 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9d9053406688cf9-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj16 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj16 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj16 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.859 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj16 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:evcclatReXiLduZaXdb7ytVvACk= 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:evcclatReXiLduZaXdb7ytVvACk='} 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e50b37df72d9422e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006eec93429fbf52be-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj17 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj17 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj17 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj17 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:iDM3AKbwvL000AjcN12/R47YSmc= 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDM3AKbwvL000AjcN12/R47YSmc='} 2024-03-04T19:09:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fbc5e4250b111fd-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030c95c5c24787bee-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj18 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj18 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj18 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.864 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj18 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:iJ4hLzKPMVCls4f59Mj5uPXkk38= 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iJ4hLzKPMVCls4f59Mj5uPXkk38='} 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2fe476bfc1041b8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099e438a444f6f672-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj19 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj19 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj19 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj19 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:97vhgY1JtC6RX8d+FrSYCAjy9ts= 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:97vhgY1JtC6RX8d+FrSYCAjy9ts='} 2024-03-04T19:09:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf2ada074ff841a8-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.868 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003d056763677a5ab-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj20 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj20 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj20 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.869 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj20 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:aGj1iS8t4P/Yw3GOAyKn+npwrhw= 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aGj1iS8t4P/Yw3GOAyKn+npwrhw='} 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007d1f8c0b4fcb2d0-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c35b797a47a3886-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj21 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj21 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj21 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj21 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Ebig/WLz2F8oTtUXgPn3bpvzetM= 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ebig/WLz2F8oTtUXgPn3bpvzetM='} 2024-03-04T19:09:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000043e5d51a3c7f40f-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.873 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9bf0ac6d3bf6c2e-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj22 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj22 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj22 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.874 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj22 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:lltJhKhfYWGVRKhtZuEmG+ZX1l8= 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:lltJhKhfYWGVRKhtZuEmG+ZX1l8='} 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005161c15c0c3552aa-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a95ce1a07c5114ce-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj23 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj23 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj23 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj23 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:p/kmNz7UOEGdk9XnyGJs5gDgAc4= 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p/kmNz7UOEGdk9XnyGJs5gDgAc4='} 2024-03-04T19:09:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7991a670ac7b3ff-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.878 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000714aa866f3c31f39-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj24 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj24 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj24 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.879 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj24 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wQtDBDZffjRCnhvjhhrpTy0DURg= 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wQtDBDZffjRCnhvjhhrpTy0DURg='} 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002581492affd0d132-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006b0e48e74174534-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj25 2024-03-04T19:09:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj25 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj25 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj25 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.882 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:yxRIMEOmM8PI7tI0CfBivaDAMcE= 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:yxRIMEOmM8PI7tI0CfBivaDAMcE='} 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aac2f1b11e234e0-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.883 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE= 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Xu0KuQ6+DXz2+tDgxL8oaExSBWE=', 'Content-Length': '0'} 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4bb3a260ce40e26-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:16 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj26 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj26 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj26 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:16 GMT 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj26 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:H/DSiX3LLS98AaGgwzYNcym5QDw= 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:16 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:H/DSiX3LLS98AaGgwzYNcym5QDw='} 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e64483029edbe48d-0065e5f484-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000001c5b868b090d7-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj27 2024-03-04T19:09:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj27 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj27 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj27 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.887 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:0AR3gJd5La8EgJb8363GB5DNvRg= 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0AR3gJd5La8EgJb8363GB5DNvRg='} 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e42acd0bdb5fca3-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.888 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d67528dff1d38ea-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj28 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj28 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj28 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj28 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:zN1xG+5jSwDSFg1RzyHd2519Wpo= 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:zN1xG+5jSwDSFg1RzyHd2519Wpo='} 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000baf2c453c4a6893b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063a86f9c7fcf0734-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj29 2024-03-04T19:09:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj29 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj29 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj29 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:V8Q25hmdDKY3aUAL2kPwH0QX2Fs= 2024-03-04T19:09:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V8Q25hmdDKY3aUAL2kPwH0QX2Fs='} 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcb4180d3460ab27-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.893 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8986ef13cd92561-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj30 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj30 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj30 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj30 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:/BMvJ4No/x0F+cDvT1vdvmg5S7o= 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/BMvJ4No/x0F+cDvT1vdvmg5S7o='} 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044dcc1395eb152b4-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8d84e8ffebd2001-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj31 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj31 2024-03-04T19:09:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj31 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj31 2024-03-04T19:09:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jZSgzLakSU35jgRO/zepL0q43oU= 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jZSgzLakSU35jgRO/zepL0q43oU='} 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be651e6447cb0709-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.898 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad87a72313bff84d-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj32 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj32 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj32 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj32 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:YSBSfrcDNxL7Mn3e0H68oASpdo0= 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:YSBSfrcDNxL7Mn3e0H68oASpdo0='} 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef52b73001aa4b7e-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d02583d65d52457f-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj33 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj33 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj33 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.903 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj33 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:mW8NaBfjLJEzokuTy5BIQUXoBBQ= 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mW8NaBfjLJEzokuTy5BIQUXoBBQ='} 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056be2cafe3d85a72-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000138c1e0959dfe0f2-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj34 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj34 2024-03-04T19:09:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj34 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj34 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I1cvtrlhxBFXpynbljfQHPKVh2s= 2024-03-04T19:09:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I1cvtrlhxBFXpynbljfQHPKVh2s='} 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c3a50cde03720ed-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.907 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ced6eec3c6629b8-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj35 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj35 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj35 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj35 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rclb1FkpEEErsRlyhFt1V0rv+e4= 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rclb1FkpEEErsRlyhFt1V0rv+e4='} 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087b5a223ac7b5f81-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c6e7b1e80fcbf6-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj36 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj36 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj36 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj36 2024-03-04T19:09:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:RFmEdUNrvAmIkPL75xpVX2B8tjI= 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RFmEdUNrvAmIkPL75xpVX2B8tjI='} 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000896358968d0ef778-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.912 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a9f3db3ae9c5595-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj37 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj37 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj37 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj37 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wyfrEzimDpGgPblxzVP/we1rBPA= 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wyfrEzimDpGgPblxzVP/we1rBPA='} 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bfaecca3db5d973-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec0c0ac6e502b6c8-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj38 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj38 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj38 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.916 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj38 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:OUxhfBWuNHc1FwcBkWhSCgaisMw= 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OUxhfBWuNHc1FwcBkWhSCgaisMw='} 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4003625a3c8a3c1-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.917 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee5241f9fb8dc6db-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj39 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj39 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj39 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj39 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TZzCoUBDvGIUh3oEJpNZ7BJIAk8= 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TZzCoUBDvGIUh3oEJpNZ7BJIAk8='} 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c351632b1b596dc-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b64bd0d47dc9e31a-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj40 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj40 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj40 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.921 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj40 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:u0vA9Shxju3NmKA//vh4ovsas+w= 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:u0vA9Shxju3NmKA//vh4ovsas+w='} 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a12e14c42bf4ff2c-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d011d68d46090ef-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj41 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj41 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj41 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj41 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:U6a96BYyFLeezg82rSUBPHtgcRw= 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:U6a96BYyFLeezg82rSUBPHtgcRw='} 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba46b50807163892-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a66384b605b03678-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj42 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj42 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj42 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.926 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj42 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2j5oWPieZi03g+FUweG96vrzq94= 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2j5oWPieZi03g+FUweG96vrzq94='} 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008495ee0f0c2da30-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc8a7ed29f12e7b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj43 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj43 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj43 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj43 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:deDPa/lZBHHpGGihPNY6tVGK+f8= 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:deDPa/lZBHHpGGihPNY6tVGK+f8='} 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cf8d53a950ae15d-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.930 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2a0421c242a3481-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj44 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj44 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj44 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.931 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj44 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:M4dWY5O+xF7Qi/4qLG/XwtDMEw4= 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M4dWY5O+xF7Qi/4qLG/XwtDMEw4='} 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcfac8c6753e6be9-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002da2624d493e835c-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj45 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj45 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj45 2024-03-04T19:09:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj45 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:dJS2MMTY8+FwxEvRYqOo9217Irs= 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dJS2MMTY8+FwxEvRYqOo9217Irs='} 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000747698855a5626cb-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.935 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5bafa6616314aab-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj46 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj46 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj46 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.936 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj46 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:9ovMiGcDLoH0XRAE4+M5DUhV40M= 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:9ovMiGcDLoH0XRAE4+M5DUhV40M='} 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3481f693aeed615-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f139df0a57e0a21b-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj47 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj47 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj47 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj47 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2xBs1QnztYesm+5UKRLmi+8L7bo= 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2xBs1QnztYesm+5UKRLmi+8L7bo='} 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062bacfcb22569f31-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f32c711808bbc481-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj48 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj48 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj48 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.941 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj48 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TO1IBJsjDvI47LwQOxMkrCESFS4= 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TO1IBJsjDvI47LwQOxMkrCESFS4='} 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a52c90e18d1fd59-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/ 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/ 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/ 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:/sbnvju-17/ 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg= 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:N9Uge1H9r/H3xP7U6iqnNwL20lg=', 'Content-Length': '0'} 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048375c65b449f682-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-17/obj49 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-17/obj49 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-17/obj49 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:/sbnvju-17/obj49 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:LjANaMIz/YXH+MURHZHBq60BvP4= 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:LjANaMIz/YXH+MURHZHBq60BvP4='} 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090435843a9a46d87-0065e5f485-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f39c0e73469b21a-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj1 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj1 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj1 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.946 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj1 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Vz6rEU6rvodg2jePO8NMBGen/0E= 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Vz6rEU6rvodg2jePO8NMBGen/0E='} 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000241a5d08ed3ae648-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.947 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a30beeba3c9ee432-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj2 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj2 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj2 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj2 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:o7/GzzWcApVcZtKXFG5TRIv+kLc= 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:o7/GzzWcApVcZtKXFG5TRIv+kLc='} 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051012e5d85129936-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5452ad68aed90c0-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj3 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj3 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj3 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj3 2024-03-04T19:09:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:FtoNZppBEl5UBRD1cKljlynW9JQ= 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:FtoNZppBEl5UBRD1cKljlynW9JQ='} 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5c4c5bebc14add9-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.952 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007efad18f8dbf0e1e-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj4 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj4 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj4 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj4 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rEq/MxCkztIwztZ2DsfU+20WT54= 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rEq/MxCkztIwztZ2DsfU+20WT54='} 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efee830ae8044a6c-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc71e69e41f39a65-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj5 2024-03-04T19:09:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj5 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj5 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj5 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.956 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5gNy/bkx9MlMuGOCypkv6y+ThCk= 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5gNy/bkx9MlMuGOCypkv6y+ThCk='} 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c12f1f5774abeaca-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.957 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002acc2281d4aaed20-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj6 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj6 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj6 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj6 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:gQ2j60OEFqcNKV6ehf1KLupIiH0= 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gQ2j60OEFqcNKV6ehf1KLupIiH0='} 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b91f58890d607dad-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f17be6ac34de7b8d-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj7 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj7 2024-03-04T19:09:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj7 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj7 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:/+Ioua+H3sX/ik3dOUtRIlFIiBo= 2024-03-04T19:09:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:/+Ioua+H3sX/ik3dOUtRIlFIiBo='} 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003908bee176354f7b-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.962 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008070562c1c93bc01-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj8 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj8 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj8 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj8 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:tYpyCK+ckbMgcPwo19fA4noOp3w= 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tYpyCK+ckbMgcPwo19fA4noOp3w='} 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d52f919f774821c4-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b91356aa657f5d-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj9 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj9 2024-03-04T19:09:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj9 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj9 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:6JFxLPhZxvSraE/pKz33OHzHOiQ= 2024-03-04T19:09:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6JFxLPhZxvSraE/pKz33OHzHOiQ='} 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f9a5179da4ec665-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.967 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015a227e1bb48852b-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj10 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj10 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj10 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.968 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj10 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:sE4fnbUt5OoSqs43KWs7NvF94Fw= 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sE4fnbUt5OoSqs43KWs7NvF94Fw='} 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008521bc343f115721-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056c2dcfacb69b125-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj11 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj11 2024-03-04T19:09:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj11 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj11 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:xnnEGhGC9NDYdxuyTFlyADSRbLk= 2024-03-04T19:09:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:xnnEGhGC9NDYdxuyTFlyADSRbLk='} 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b516cbdd82ec0d77-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.972 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000233177e1fca5f4b9-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj12 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj12 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj12 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.973 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj12 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wRgEhuH5D983BZmT1tIEXZ674eE= 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wRgEhuH5D983BZmT1tIEXZ674eE='} 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb648211b99feeac-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14= 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:O4cHR7AENSeLKt7hS6g9SgDSD14=', 'Content-Length': '0'} 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb17969e06e826f3-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:17 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj13 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj13 2024-03-04T19:09:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj13 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:17 GMT 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj13 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.976 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:99h1ak2SH2JgPhEOjjkPk9yLMVo= 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:17 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:99h1ak2SH2JgPhEOjjkPk9yLMVo='} 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c763682f8118c2e-0065e5f485-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.977 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000755eb979664c8a41-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj14 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj14 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj14 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.978 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj14 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:MvFLcqj8W/c0lK60uMcqnzsoSzM= 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MvFLcqj8W/c0lK60uMcqnzsoSzM='} 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f664d8a0d40bbce-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000481709a52099893c-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj15 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj15 2024-03-04T19:09:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj15 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj15 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Kx6VC48ss9L6r6EOVP4sWYssApk= 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Kx6VC48ss9L6r6EOVP4sWYssApk='} 2024-03-04T19:09:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe353b9c38e827a0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71f3755e85ad07a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj16 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj16 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj16 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.983 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj16 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Okx0GtoJtWnnUk0r6Xcf0vFoUCc= 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Okx0GtoJtWnnUk0r6Xcf0vFoUCc='} 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094344a59992badee-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000351bc945120736b7-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj17 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj17 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj17 2024-03-04T19:09:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj17 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:l7V1qjsiS2WCO5bXaLLV6wAGcVU= 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:l7V1qjsiS2WCO5bXaLLV6wAGcVU='} 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc50b7b93d5bd679-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.987 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd98983824045d9f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj18 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj18 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj18 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.988 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj18 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:qAV3phT6mjoGBBojGdu3+0zwkw0= 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qAV3phT6mjoGBBojGdu3+0zwkw0='} 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1a3e865d95074f3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008deccc513304ab1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj19 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj19 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj19 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj19 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:gtc65Zq5Bg7OYOOUDwjBbzmWII0= 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:gtc65Zq5Bg7OYOOUDwjBbzmWII0='} 2024-03-04T19:09:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aecf5f30a5cdbb2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e2b1af94744dfcf-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj20 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj20 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj20 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.993 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj20 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wSlQQHgIUTfTqmdnGa0Tw2lLYhA= 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wSlQQHgIUTfTqmdnGa0Tw2lLYhA='} 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd10427098df723-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2b57717e4682760-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj21 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj21 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj21 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj21 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5l7qMUoZdFPuF55WoZjpzvTPV9g= 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5l7qMUoZdFPuF55WoZjpzvTPV9g='} 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049b6bc2ed11f8325-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bedf155154743fc-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj22 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj22 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj22 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:55.998 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj22 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZnT/GJp9zqw8BCqedcBPzvShpgo= 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZnT/GJp9zqw8BCqedcBPzvShpgo='} 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b648164d972195-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcfd0018186a8f3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj23 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj23 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj23 2024-03-04T19:09:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj23 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IxZW3y0oqQ36fwGqX/brWrHphc0= 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IxZW3y0oqQ36fwGqX/brWrHphc0='} 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e67cc168770c2043-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081039a62a30d57d9-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj24 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj24 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj24 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.003 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj24 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:dG9NhdcBSUgG+nBaBLcQxv3paqQ= 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:dG9NhdcBSUgG+nBaBLcQxv3paqQ='} 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016beb55029e8f3c0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000978aff216ebe4b2b-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj25 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj25 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj25 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj25 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TZLjbfuk3uBw/1cPbP3TDeyYegI= 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TZLjbfuk3uBw/1cPbP3TDeyYegI='} 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a96d3dade0269946-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee356aa831089309-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj26 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj26 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj26 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj26 2024-03-04T19:09:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2yFCAsg/b7Qg0lcLJ5XE3BnbeWw= 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2yFCAsg/b7Qg0lcLJ5XE3BnbeWw='} 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000223f3cd34268346a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.009 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000623a73684df5410f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj27 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj27 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj27 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj27 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:XCfoafgARPiS/NjnmEbvtU4mGgM= 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XCfoafgARPiS/NjnmEbvtU4mGgM='} 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d6a2c56b5637119-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4f4b99cc9bc694f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj28 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj28 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj28 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj28 2024-03-04T19:09:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:SmVgqW05TzJIPNr9VPDy87CM8II= 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SmVgqW05TzJIPNr9VPDy87CM8II='} 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e9cc83a33db7d8-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.014 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff53156e51ff9579-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj29 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj29 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj29 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj29 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:EXGo1FF95svt9jegJ1H+LG2vlDw= 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EXGo1FF95svt9jegJ1H+LG2vlDw='} 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1a8fea8152cfc1b-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5a8f6fd3963352-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj30 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj30 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj30 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj30 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.018 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:S//ieUp+uELlCj6kO18UJvdoeB4= 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S//ieUp+uELlCj6kO18UJvdoeB4='} 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faf8f0ab6f2127a4-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.019 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de257ad6e4b4fb80-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj31 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj31 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj31 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj31 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.021 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:EYn/i/lc42GBqcV/KHpzJh3cFXs= 2024-03-04T19:09:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EYn/i/lc42GBqcV/KHpzJh3cFXs='} 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020c7a2d52208b57d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.023 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020a31df6677c2fe1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj32 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj32 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj32 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj32 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:odNQOb0+dJ7me6ESNsc0XkRa5MM= 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:odNQOb0+dJ7me6ESNsc0XkRa5MM='} 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0976a29871aaed2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c17526cedebc3a1d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj33 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj33 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj33 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj33 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:n/fD0jr1lDDBwgWhAWs0gxlXFSw= 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:n/fD0jr1lDDBwgWhAWs0gxlXFSw='} 2024-03-04T19:09:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003683ced434f14224-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8a04534c1c19f6c-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj34 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj34 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj34 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.029 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj34 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:XGU73oSBGV6xuPxrf6uLIqivwd4= 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XGU73oSBGV6xuPxrf6uLIqivwd4='} 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe145d285d927df6-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7b965409a6aa37f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj35 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj35 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj35 2024-03-04T19:09:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj35 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:icdSXG0hOyiXPLsufL9CnOioQEQ= 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:icdSXG0hOyiXPLsufL9CnOioQEQ='} 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000084afb26c43196c8-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034c5645ac36a4cbb-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj36 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj36 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj36 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.034 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj36 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:SvG9tZ06VqXPe/Xbh+QLh++RAfQ= 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:SvG9tZ06VqXPe/Xbh+QLh++RAfQ='} 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c0d19e4f00a5bad-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d3d6125fca73a2-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj37 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj37 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj37 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.037 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj37 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:uWrkCOOCYBEyTlITtLrnkcdBd3k= 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uWrkCOOCYBEyTlITtLrnkcdBd3k='} 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2c0de123d661b55-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af36e40dbc939550-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj38 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj38 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj38 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj38 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:7dFUP8mYtrih5BY+kTAWFRSTPU0= 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:7dFUP8mYtrih5BY+kTAWFRSTPU0='} 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043b5a0b183bc93db-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001610e76087499bad-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj39 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj39 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj39 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.042 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj39 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rnETUFYZlMvzDpu8hoyRnz7FE1k= 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rnETUFYZlMvzDpu8hoyRnz7FE1k='} 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058ff2d274ddfa5cd-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8c50f6cffdeab58-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj40 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj40 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj40 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj40 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:+QQ2NL8PZYUyO9avhLTdJ4TXoQg= 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+QQ2NL8PZYUyO9avhLTdJ4TXoQg='} 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000befbf20221ae0d50-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092dd1c1119493219-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj41 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj41 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj41 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.047 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj41 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:UeBQ140qJnkp+9awYntKh7F91Jc= 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:UeBQ140qJnkp+9awYntKh7F91Jc='} 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec929a21674376fb-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003283f65521336aae-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj42 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj42 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj42 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj42 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:tlIsfY17t+pl7bLnH35/WxDTyXA= 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:tlIsfY17t+pl7bLnH35/WxDTyXA='} 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094057f10999e8fa4-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a6845cb7dddb93f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj43 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj43 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj43 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.052 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj43 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:PzZHP4Shxg/s/D4GyZgFwW9l/Dg= 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PzZHP4Shxg/s/D4GyZgFwW9l/Dg='} 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a72fc56cc5d84f8a-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6fd05d33f22670f-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj44 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj44 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj44 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj44 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:JIdll/AQBKmprpTpHe7Ja4npEIg= 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:JIdll/AQBKmprpTpHe7Ja4npEIg='} 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e455b697dd3a0aa3-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.056 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2bb27d72f7ff1ca-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj45 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj45 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj45 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.057 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj45 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:CjGGcwyY4nkuK6xPWSGz7DMqu+o= 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CjGGcwyY4nkuK6xPWSGz7DMqu+o='} 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047c7df1586efccf0-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018eefd01008d0083-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj46 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj46 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj46 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj46 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:RF5bvjuzS7BXJPfE/BEAXIhjR3w= 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:RF5bvjuzS7BXJPfE/BEAXIhjR3w='} 2024-03-04T19:09:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a07068af66128157-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.061 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c530f1990bcb829-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj47 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj47 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj47 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj47 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:vKA6etbYid0oWSeUqLfJvgRHSlg= 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vKA6etbYid0oWSeUqLfJvgRHSlg='} 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000371a95ac8847452e-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000394d9d18c7d3f24d-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj48 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj48 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj48 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.065 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj48 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ZAoYtZQ8mL8mIvSiHPUse1f+nCg= 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ZAoYtZQ8mL8mIvSiHPUse1f+nCg='} 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003797e2294aa358d1-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/ 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/ 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/ 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.066 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:/sbnvju-18/ 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0= 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:I7JPvab0VmRb66NbgrN6faVYGi0=', 'Content-Length': '0'} 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb87cef1942b2a09-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-18/obj49 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-18/obj49 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-18/obj49 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:19:18 GMT 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:/sbnvju-18/obj49 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:z5k1E4aVHjTYke7olWmOfBV/CLM= 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:19:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:z5k1E4aVHjTYke7olWmOfBV/CLM='} 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee75e57fc2885c16-0065e5f486-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:19:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709569155.099706_86.1","last_update":"2024-03-04T16:19:15.099706Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:19.098+0000 7f5e5711f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:19.099+0000 7f5e5711f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569155.099706_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:19:15.099706Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: 2024-03-04T16:19:19.214+0000 7f6cd2813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: 2024-03-04T16:19:19.215+0000 7f6cd2813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709569155.099706_86.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:19:15.099706Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:19:19Z"} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709569155.099706_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:19:15.099706Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709569155.099706_86.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'sbnvju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket sbnvju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:19.337+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:24.338+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:29.338+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:34.339+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:39.340+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:44.340+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:49.341+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:54.342+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:19:59.343+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:04.344+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:09.344+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:14.345+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:19.345+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:24.346+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:29.347+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:34.348+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:39.348+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:44.349+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:49.350+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:54.351+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:20:59.352+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:04.351+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:09.352+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:14.353+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:19.353+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:24.354+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:29.355+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:34.356+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:39.356+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:44.357+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:49.358+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:54.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:21:59.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:04.359+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:09.360+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:14.360+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:19.361+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:24.362+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:29.362+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:34.363+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:39.364+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:44.364+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:49.365+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:54.365+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:22:59.366+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:04.367+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:09.368+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:14.368+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:19.369+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:24.369+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:29.370+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:34.371+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:39.371+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:44.373+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:49.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:54.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:23:59.374+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:24:04.375+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:24:09.375+0000 7fe4d6e16800 1 waiting to reach incremental sync.. 2024-03-04T19:09:56.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:24:14.375+0000 7fe4d6e16800 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:24:14.375+0000 7fe4d6e16800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:24:14.375+0000 7fe4d6e16800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-03-04T19:09:56.075 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 903, in test_bucket_delete_notempty 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-39 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-39/ 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-39/ 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-39/ 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:39:32 GMT 2024-03-04T19:09:56.077 INFO:tasks.rgw_multisite_tests:/sbnvju-39/ 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:njY7fQZ7hrtv6x9NwUhR1T3/hqg= 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:njY7fQZ7hrtv6x9NwUhR1T3/hqg=', 'Content-Length': '0'} 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000df1b100fd70e49f6-0065e5f944-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-40 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-40/ 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-40/ 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-40/ 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.078 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:39:32 GMT 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:/sbnvju-40/ 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IDCxGACbsbOy3+ceit0WCPH+9OE= 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IDCxGACbsbOy3+ceit0WCPH+9OE=', 'Content-Length': '0'} 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005e3b3f910ea93bd0-0065e5f944-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-41 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-41/ 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-41/ 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-41/ 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:39:32 GMT 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:/sbnvju-41/ 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rcFOlF05vUqPmBJWvb8hZwOF1K0= 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:39:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rcFOlF05vUqPmBJWvb8hZwOF1K0=', 'Content-Length': '0'} 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f2fe79137587c585-0065e5f944-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709570372.221520_469.1","last_update":"2024-03-04T16:39:32.221520Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:32.488+0000 7f5c84a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:32.489+0000 7f5c84a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:32Z"} 2024-03-04T19:09:56.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:32Z"} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:37.599+0000 7f608eec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:37.600+0000 7f608eec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:37Z"} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:37Z"} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:42.710+0000 7f568cf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:42.711+0000 7f568cf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:42Z"} 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:42Z"} 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:47.823+0000 7f41a4f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:47.824+0000 7f41a4f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:47Z"} 2024-03-04T19:09:56.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:47Z"} 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:52.993+0000 7fd30b34b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:52.994+0000 7fd30b34b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:52Z"} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:52Z"} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:58.110+0000 7f7748730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:39:58.111+0000 7f7748730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:58Z"} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:39:58Z"} 2024-03-04T19:09:56.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:03.222+0000 7ff6cf814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:03.223+0000 7ff6cf814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:03Z"} 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:03Z"} 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:08.337+0000 7f747873b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:08.338+0000 7f747873b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:08Z"} 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:08Z"} 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:13.526+0000 7fe5b7444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:13.527+0000 7fe5b7444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:13Z"} 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:13Z"} 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:18.642+0000 7fab54922800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:18.643+0000 7fab54922800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:18Z"} 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:18Z"} 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:23.765+0000 7fa7b6439800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:23.766+0000 7fa7b6439800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:23Z"} 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:23Z"} 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:28.890+0000 7fb18a012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:28.891+0000 7fb18a012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:28Z"} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:28Z"} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:34.017+0000 7f5075faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:34.018+0000 7f5075faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:34Z"} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:34Z"} 2024-03-04T19:09:56.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.819018_429.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.819018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.819018_429.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.819018_429.1 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:39.180+0000 7f2253c42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:39.181+0000 7f2253c42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:39Z"} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:39Z"} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.838183_430.1 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:44.293+0000 7ffb18010800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:44.294+0000 7ffb18010800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:44Z"} 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:44Z"} 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:49.408+0000 7fe705326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:49.409+0000 7fe705326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:49Z"} 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:49Z"} 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:54.523+0000 7fc1ea711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:54.524+0000 7fc1ea711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:54Z"} 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:54Z"} 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.816633_428.1 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:59.646+0000 7fefca7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:40:59.647+0000 7fefca7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:59Z"} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:40:59Z"} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.662426_417.1 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:04.760+0000 7f5761b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:04.761+0000 7f5761b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:04Z"} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:04Z"} 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:09.872+0000 7ff4f3232800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:09.873+0000 7ff4f3232800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:09Z"} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:09Z"} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:14.984+0000 7fa95c9af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:14.985+0000 7fa95c9af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:14Z"} 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:14Z"} 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:20.118+0000 7f461b224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:20.119+0000 7f461b224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:20Z"} 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:20Z"} 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:25.241+0000 7fad49011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:25.242+0000 7fad49011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:25Z"} 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:25Z"} 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:30.429+0000 7f4b19a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:30.430+0000 7f4b19a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:30Z"} 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:30Z"} 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:35.540+0000 7f50d4024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:35.541+0000 7f50d4024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:35Z"} 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:35Z"} 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.715026_419.1 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:40.660+0000 7fb1aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:40.661+0000 7fb1aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:40Z"} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:40Z"} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:45.784+0000 7fe97fc9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:45.785+0000 7fe97fc9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:45Z"} 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:45Z"} 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:50.909+0000 7f0367f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:50.910+0000 7f0367f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:50Z"} 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:50Z"} 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:56.034+0000 7fd06868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:41:56.035+0000 7fd06868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:56Z"} 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:41:56Z"} 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:01.161+0000 7f15dc20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:01.162+0000 7f15dc20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:01Z"} 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:01Z"} 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.738966_421.1 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:06.275+0000 7f6c7a40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:06.276+0000 7f6c7a40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:06Z"} 2024-03-04T19:09:56.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:06Z"} 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:11.389+0000 7f84e8e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:11.389+0000 7f84e8e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:11Z"} 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:11Z"} 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:16.502+0000 7f189713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:16.503+0000 7f189713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:16Z"} 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:16Z"} 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:21.626+0000 7fad53c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:21.627+0000 7fad53c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:21Z"} 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:21Z"} 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:26.752+0000 7fe101824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:26.753+0000 7fe101824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:26Z"} 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:26Z"} 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:31.880+0000 7f649631e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:31.881+0000 7f649631e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:31Z"} 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:31Z"} 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:37.030+0000 7fa434b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:37.031+0000 7fa434b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:37Z"} 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:37Z"} 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:42.142+0000 7f0231a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:42.142+0000 7f0231a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:42Z"} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:42Z"} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:47.259+0000 7fde4cc85800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:47.260+0000 7fde4cc85800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:47Z"} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:47Z"} 2024-03-04T19:09:56.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:52.383+0000 7fc4c6b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:52.384+0000 7fc4c6b3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:52Z"} 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:52Z"} 2024-03-04T19:09:56.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:57.506+0000 7fb698f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:42:57.507+0000 7fb698f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:57Z"} 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:42:57Z"} 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:02.661+0000 7f5582ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:02.661+0000 7f5582ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:02Z"} 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:02Z"} 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:07.776+0000 7f663ff3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:07.777+0000 7f663ff3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:07Z"} 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:07Z"} 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:12.895+0000 7f191d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:12.896+0000 7f191d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:12Z"} 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:12Z"} 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:18.018+0000 7f28b2412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:18.019+0000 7f28b2412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:18Z"} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:18Z"} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:23.174+0000 7fe1fe6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:23.174+0000 7fe1fe6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:23Z"} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:23Z"} 2024-03-04T19:09:56.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:28.289+0000 7fe2ad726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:28.290+0000 7fe2ad726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:28Z"} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:28Z"} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:33.401+0000 7fcce0230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:33.402+0000 7fcce0230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:33Z"} 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:33Z"} 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:38.515+0000 7faefe730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:38.516+0000 7faefe730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:38Z"} 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:38Z"} 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:43.640+0000 7f858413c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:43.641+0000 7f858413c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:43Z"} 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:43Z"} 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:48.787+0000 7f5fa412a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:48.788+0000 7f5fa412a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:48Z"} 2024-03-04T19:09:56.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:48Z"} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.815644_427.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.815644Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.815644_427.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.815644_427.1 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:53.983+0000 7f050b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:53.984+0000 7f050b68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:53Z"} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:53Z"} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:59.097+0000 7f449fb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:43:59.098+0000 7f449fb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:59Z"} 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:43:59Z"} 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:04.209+0000 7fb2e4f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:04.210+0000 7fb2e4f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:04Z"} 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:04Z"} 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:09.333+0000 7f233c32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:09.334+0000 7f233c32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:09Z"} 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:09Z"} 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:14.454+0000 7f49e77af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:14.455+0000 7f49e77af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:14Z"} 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:14Z"} 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.662426_417.1 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:19.580+0000 7f7de8c20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:19.581+0000 7f7de8c20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:19Z"} 2024-03-04T19:09:56.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:19Z"} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:24.748+0000 7fd3e9e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:24.749+0000 7fd3e9e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:24Z"} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:24Z"} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:29.863+0000 7fc5d6a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:29.864+0000 7fc5d6a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:29Z"} 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:29Z"} 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:34.976+0000 7fe017617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:34.977+0000 7fe017617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:34Z"} 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:34Z"} 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570372.221520_469.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.133 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 936, in test_multi_period_incremental_sync 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-42 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-42/ 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-42/ 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-42/ 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:44:39 GMT 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:/sbnvju-42/ 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:vjlv8lW59s5m9USSrLigp5JjxqY= 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:39 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vjlv8lW59s5m9USSrLigp5JjxqY=', 'Content-Length': '0'} 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000079b5cee737700a81-0065e5fa78-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-43 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-43/ 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-43/ 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-43/ 2024-03-04T19:09:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:44:40 GMT 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:/sbnvju-43/ 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:oY0QyuqSlHy133iNVpfzhClbebY= 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oY0QyuqSlHy133iNVpfzhClbebY=', 'Content-Length': '0'} 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000acd1b1fd42b4a896-0065e5fa78-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-44 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-44/ 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-44/ 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-44/ 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.138 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:44:40 GMT 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:/sbnvju-44/ 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Zr/X+YXo5tX5rxrpCXXd7CyYZ/Y= 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:44:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Zr/X+YXo5tX5rxrpCXXd7CyYZ/Y=', 'Content-Length': '0'} 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007957b4da74e7db5e-0065e5fa78-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:44:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709570680.085880_523.1","last_update":"2024-03-04T16:44:40.085880Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:40.321+0000 7f6f9dd0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:40.322+0000 7f6f9dd0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:40Z"} 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:40Z"} 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:45.514+0000 7f4e1cd2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:45.516+0000 7f4e1cd2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:45Z"} 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:45Z"} 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:50.683+0000 7fc71051f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:50.684+0000 7fc71051f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:50Z"} 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:50Z"} 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:55.795+0000 7f260211f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:44:55.796+0000 7f260211f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:55Z"} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:44:55Z"} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:00.909+0000 7f9654d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:00.910+0000 7f9654d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:00Z"} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:00Z"} 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:06.091+0000 7fbd3042d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:06.092+0000 7fbd3042d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:06Z"} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:06Z"} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:11.283+0000 7f5921e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:11.284+0000 7f5921e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:11Z"} 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:11Z"} 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:16.398+0000 7f4ba3f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:16.398+0000 7f4ba3f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:16Z"} 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:16Z"} 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.816633_428.1 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:21.510+0000 7f4474412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:21.511+0000 7f4474412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:21Z"} 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:21Z"} 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:26.624+0000 7f16742c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:26.625+0000 7f16742c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:26Z"} 2024-03-04T19:09:56.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:26Z"} 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.801670_425.1 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:31.756+0000 7f46bfb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:31.757+0000 7f46bfb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:31Z"} 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:31Z"} 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:36.873+0000 7ff002b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:36.874+0000 7ff002b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:36Z"} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:36Z"} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:42.017+0000 7fc380a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:42.018+0000 7fc380a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:42Z"} 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.662426_417.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.662426Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:42Z"} 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.662426_417.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.662426Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.662426_417.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.662426_417.1 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:47.132+0000 7f9ed1c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:47.132+0000 7f9ed1c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:47Z"} 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:47Z"} 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:52.251+0000 7fd0de131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:52.252+0000 7fd0de131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:52Z"} 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:52Z"} 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:57.435+0000 7f9b96814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:45:57.436+0000 7f9b96814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:57Z"} 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:45:57Z"} 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:02.557+0000 7f73784c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:02.558+0000 7f73784c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:02Z"} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:02Z"} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:07.672+0000 7fe1cd824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:07.673+0000 7fe1cd824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:07Z"} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:07Z"} 2024-03-04T19:09:56.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:12.796+0000 7f030ae12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:12.797+0000 7f030ae12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:12Z"} 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:12Z"} 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:17.913+0000 7f4cd5824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:17.914+0000 7f4cd5824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:17Z"} 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:17Z"} 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:23.037+0000 7f1665311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:23.038+0000 7f1665311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:23Z"} 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:23Z"} 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:28.163+0000 7fd27fb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:28.164+0000 7fd27fb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:28Z"} 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:28Z"} 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:33.288+0000 7f9d6ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:33.289+0000 7f9d6ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:33Z"} 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:33Z"} 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:38.504+0000 7ff8fdd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:38.505+0000 7ff8fdd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:38Z"} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:38Z"} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:43.681+0000 7fae6a02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:43.682+0000 7fae6a02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:43Z"} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:43Z"} 2024-03-04T19:09:56.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:48.798+0000 7f86ce22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:48.799+0000 7f86ce22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:48Z"} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:48Z"} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:53.929+0000 7f7b70230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:53.930+0000 7f7b70230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:53Z"} 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:53Z"} 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:59.043+0000 7f4573b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:46:59.044+0000 7f4573b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:59Z"} 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:46:59Z"} 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:04.155+0000 7f2bd6a8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:04.156+0000 7f2bd6a8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:04Z"} 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:04Z"} 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:09.285+0000 7f13a8b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:09.285+0000 7f13a8b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:09Z"} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:09Z"} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:14.410+0000 7f420b737800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:14.411+0000 7f420b737800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:14Z"} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:14Z"} 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:19.589+0000 7f99bc067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:19.590+0000 7f99bc067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:19Z"} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:19Z"} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:24.783+0000 7f6598c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:24.794+0000 7f6598c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:24Z"} 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:24Z"} 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:29.963+0000 7f20cd124800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:29.964+0000 7f20cd124800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:29Z"} 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:29Z"} 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:35.086+0000 7f284ed31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:35.087+0000 7f284ed31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:35Z"} 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:35Z"} 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:40.212+0000 7f05b5832800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:40.213+0000 7f05b5832800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:40Z"} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:40Z"} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:45.328+0000 7f82fd08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:45.329+0000 7f82fd08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:45Z"} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:45Z"} 2024-03-04T19:09:56.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:50.453+0000 7f222d93b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:50.454+0000 7f222d93b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:50Z"} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:50Z"} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:55.596+0000 7f96b8ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:47:55.597+0000 7f96b8ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:55Z"} 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:47:55Z"} 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:00.726+0000 7f51cac2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:00.727+0000 7f51cac2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:00Z"} 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:00Z"} 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:05.870+0000 7f0aee40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:05.871+0000 7f0aee40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:05Z"} 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:05Z"} 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:11.005+0000 7f6ea8c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:11.006+0000 7f6ea8c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:11Z"} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:11Z"} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:16.217+0000 7f09b5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:16.218+0000 7f09b5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:16Z"} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:16Z"} 2024-03-04T19:09:56.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:21.353+0000 7f3bbfe8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:21.354+0000 7f3bbfe8a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:21Z"} 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:21Z"} 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:26.474+0000 7f9ec1111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:26.475+0000 7f9ec1111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:26Z"} 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:26Z"} 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:31.606+0000 7f5c35015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:31.607+0000 7f5c35015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:31Z"} 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:31Z"} 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:36.748+0000 7f9474742800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:36.749+0000 7f9474742800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:36Z"} 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:36Z"} 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.798294_423.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.798294Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.798294_423.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.798294_423.1 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:41.893+0000 7fcff49af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:41.893+0000 7fcff49af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:41Z"} 2024-03-04T19:09:56.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:41Z"} 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:47.036+0000 7fc9cdf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:47.037+0000 7fc9cdf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:47Z"} 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:47Z"} 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:52.171+0000 7f26f7324800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:52.172+0000 7f26f7324800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:52Z"} 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:52Z"} 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:57.361+0000 7f68d4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:48:57.362+0000 7f68d4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:57Z"} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:48:57Z"} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:02.482+0000 7f0f5a13b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:02.483+0000 7f0f5a13b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:02Z"} 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:02Z"} 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:07.597+0000 7fba32122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:07.598+0000 7fba32122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:07Z"} 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:07Z"} 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:12.722+0000 7f22cf7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:12.723+0000 7f22cf7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:12Z"} 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:12Z"} 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:17.859+0000 7ff9cb830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:17.860+0000 7ff9cb830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:17Z"} 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:17Z"} 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:22.993+0000 7f92eed11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:22.994+0000 7f92eed11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:22Z"} 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:22Z"} 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:28.123+0000 7f12b993b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:28.124+0000 7f12b993b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:28Z"} 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:28Z"} 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:33.261+0000 7fecf0b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:33.262+0000 7fecf0b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:33Z"} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:33Z"} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:38.415+0000 7fc8f7fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:38.416+0000 7fc8f7fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:38Z"} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:38Z"} 2024-03-04T19:09:56.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:43.561+0000 7f41e7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:43.562+0000 7f41e7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:43Z"} 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:43Z"} 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570680.085880_523.1 target=1_1709570369.799310_424.1 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.191 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1021, in test_datalog_autotrim 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-45 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-45/ 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-45/ 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-45/ 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:/sbnvju-45/ 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ey3kEGarYfskZiPhWj/WUjpZqnY= 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ey3kEGarYfskZiPhWj/WUjpZqnY=', 'Content-Length': '0'} 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a06a1596a7b8f2bc-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-46 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-46/ 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-46/ 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-46/ 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.194 INFO:tasks.rgw_multisite_tests:/sbnvju-46/ 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:J/uSG4lXjAFEkNWf2LScx0M+pR0= 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:J/uSG4lXjAFEkNWf2LScx0M+pR0=', 'Content-Length': '0'} 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007b5b37f0d6004570-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-47 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-47/ 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-47/ 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-47/ 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:/sbnvju-47/ 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:BANL2whvC2cooIw8JdPAINxgQMY= 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BANL2whvC2cooIw8JdPAINxgQMY=', 'Content-Length': '0'} 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002f2adb32108ea621-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-45/ 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-45/ 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-45/ 2024-03-04T19:09:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:/sbnvju-45/ 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:p62tCX6TSDrdpSvvdvg7wmsSDWE= 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:p62tCX6TSDrdpSvvdvg7wmsSDWE=', 'Content-Length': '0'} 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d034b72e28fce359-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-45/key 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-45/key 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-45/key 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:/sbnvju-45/key 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.198 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:BErwpOVkRokOHuRI//Ullm2brEI= 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:BErwpOVkRokOHuRI//Ullm2brEI='} 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009864630a6bb56864-0065e5fbac-4341-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-46/ 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-46/ 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-46/ 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.199 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:/sbnvju-46/ 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:36K4jKrqSho7/C16LolxuKdMrDQ= 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:36K4jKrqSho7/C16LolxuKdMrDQ=', 'Content-Length': '0'} 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064de92858d525acc-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-46/key 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-46/key 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-46/key 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:/sbnvju-46/key 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:M51kz33om5g8cjSMuZ64Yo8xGB0= 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:M51kz33om5g8cjSMuZ64Yo8xGB0='} 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003754ca8ce2296fcf-0065e5fbac-4307-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-47/ 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-47/ 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-47/ 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:/sbnvju-47/ 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:S3ikBNw0UEFs9U2RZxOmPFPpF+w= 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:S3ikBNw0UEFs9U2RZxOmPFPpF+w=', 'Content-Length': '0'} 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b29d12d1820b6975-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-47/key 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-47/key 2024-03-04T19:09:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-47/key 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:49:48 GMT 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:/sbnvju-47/key 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KFDMWCYkP+ZtFbEAq7voJBNay0E= 2024-03-04T19:09:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:49:48 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KFDMWCYkP+ZtFbEAq7voJBNay0E='} 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050ccbb2b1ef674e7-0065e5fbac-4348-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:49:48 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709570988.692350_570.1","last_update":"2024-03-04T16:49:48.692350Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:49.015+0000 7fd334d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:49.016+0000 7fd334d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:49Z"} 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:49Z"} 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:54.157+0000 7f916b610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:54.158+0000 7f916b610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:54Z"} 2024-03-04T19:09:56.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:54Z"} 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:59.303+0000 7fb6e3639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:49:59.304+0000 7fb6e3639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:59Z"} 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:49:59Z"} 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:04.430+0000 7f4476b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:04.431+0000 7f4476b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:04Z"} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:04Z"} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:09.548+0000 7ff4c8a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:09.549+0000 7ff4c8a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:09Z"} 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:09Z"} 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:14.672+0000 7f6a67022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:14.673+0000 7f6a67022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:14Z"} 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:14Z"} 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:19.832+0000 7f48a1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:19.833+0000 7f48a1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:19Z"} 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:19Z"} 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:24.964+0000 7f1ca8b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:24.965+0000 7f1ca8b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:24Z"} 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:24Z"} 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:30.105+0000 7fa6e7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:30.108+0000 7fa6e7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:30Z"} 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:30Z"} 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:35.309+0000 7fdd78636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:35.310+0000 7fdd78636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:35Z"} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:35Z"} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:40.462+0000 7f25df616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:40.463+0000 7f25df616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:40Z"} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:40Z"} 2024-03-04T19:09:56.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:45.605+0000 7f2b7d067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:45.606+0000 7f2b7d067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:45Z"} 2024-03-04T19:09:56.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:45Z"} 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.738966_421.1 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:50.727+0000 7f82e482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:50.728+0000 7f82e482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:50Z"} 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:50Z"} 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:55.878+0000 7fc02e288800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:50:55.879+0000 7fc02e288800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:55Z"} 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:50:55Z"} 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:01.032+0000 7feb11b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:01.032+0000 7feb11b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:01Z"} 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:01Z"} 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:06.163+0000 7fd1fc1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:06.164+0000 7fd1fc1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:06Z"} 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:06Z"} 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:11.297+0000 7f4841e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:11.298+0000 7f4841e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:11Z"} 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:11Z"} 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:16.496+0000 7f7143824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:16.498+0000 7f7143824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:16Z"} 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:16Z"} 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:21.705+0000 7f98fe733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:21.710+0000 7f98fe733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:21Z"} 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:21Z"} 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:26.859+0000 7fdccc424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:26.860+0000 7fdccc424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:26Z"} 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:26Z"} 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.801670_425.1 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:31.992+0000 7f163973b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:31.993+0000 7f163973b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:31Z"} 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:31Z"} 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:37.130+0000 7f046c01b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:37.131+0000 7f046c01b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:37Z"} 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.798294_423.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.798294Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:37Z"} 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.798294_423.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.798294Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.798294_423.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.798294_423.1 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:42.248+0000 7f6e2428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:42.249+0000 7f6e2428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:42Z"} 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:42Z"} 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:47.368+0000 7f622e631800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:47.369+0000 7f622e631800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:47Z"} 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:47Z"} 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:52.488+0000 7fc0f5d21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:52.489+0000 7fc0f5d21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:52Z"} 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:52Z"} 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:57.610+0000 7fcc83a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:51:57.611+0000 7fcc83a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:57Z"} 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.620918_410.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.620918Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:51:57Z"} 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.620918_410.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.620918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.620918_410.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.620918_410.1 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:02.730+0000 7fa773013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:02.731+0000 7fa773013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:02Z"} 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:02Z"} 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.838183_430.1 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:07.864+0000 7f04dd131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:07.865+0000 7f04dd131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:07Z"} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:07Z"} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:13.008+0000 7fc235613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:13.009+0000 7fc235613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:13Z"} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:13Z"} 2024-03-04T19:09:56.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:18.150+0000 7f9005d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:18.151+0000 7f9005d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:18Z"} 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:18Z"} 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:23.357+0000 7fced6331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:23.358+0000 7fced6331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:23Z"} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:23Z"} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:28.525+0000 7fe103930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:28.526+0000 7fe103930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:28Z"} 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:28Z"} 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:33.657+0000 7f4c7df2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:33.658+0000 7f4c7df2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:33Z"} 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:33Z"} 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:38.794+0000 7eff6868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:38.795+0000 7eff6868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:38Z"} 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:38Z"} 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:43.927+0000 7f29bf010800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:43.928+0000 7f29bf010800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:43Z"} 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:43Z"} 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:49.064+0000 7f7cc173c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:49.065+0000 7f7cc173c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:49Z"} 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:49Z"} 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:54.184+0000 7fb599fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:54.185+0000 7fb599fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:54Z"} 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:54Z"} 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.816633_428.1 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:59.311+0000 7f862ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:52:59.312+0000 7f862ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:59Z"} 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:52:59Z"} 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:04.531+0000 7f3338729800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:04.532+0000 7f3338729800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:04Z"} 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:04Z"} 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:09.668+0000 7eff4c616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:09.669+0000 7eff4c616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:09Z"} 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:09Z"} 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:14.793+0000 7f479cc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:14.794+0000 7f479cc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:14Z"} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:14Z"} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:19.932+0000 7f56dde11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:19.933+0000 7f56dde11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:19Z"} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:19Z"} 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:25.070+0000 7f6009f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:25.071+0000 7f6009f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:25Z"} 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:25Z"} 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:30.199+0000 7f1218e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:30.200+0000 7f1218e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:30Z"} 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.601792_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.601792Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:30Z"} 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.601792_405.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.601792Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.601792_405.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.601792_405.1 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:35.335+0000 7f12a4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:35.336+0000 7f12a4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:35Z"} 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.801670_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.801670Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:35Z"} 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.801670_425.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.801670Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.801670_425.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.801670_425.1 2024-03-04T19:09:56.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:40.475+0000 7f37ca7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:40.476+0000 7f37ca7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:40Z"} 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.619960_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.619960Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:40Z"} 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.619960_409.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.619960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.619960_409.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.619960_409.1 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:45.625+0000 7fcfe0644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:45.626+0000 7fcfe0644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:45Z"} 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:45Z"} 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:50.761+0000 7f171fdaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:50.762+0000 7f171fdaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:50Z"} 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:50Z"} 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:55.893+0000 7fd82a644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:53:55.894+0000 7fd82a644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:55Z"} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:53:55Z"} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:01.022+0000 7f05acb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:01.023+0000 7f05acb31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:01Z"} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:01Z"} 2024-03-04T19:09:56.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:06.143+0000 7fbf36444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:06.144+0000 7fbf36444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:06Z"} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:06Z"} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:11.276+0000 7fe0d2a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:11.277+0000 7fe0d2a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:11Z"} 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:11Z"} 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:16.418+0000 7f012b234800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:16.419+0000 7f012b234800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:16Z"} 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.605133_407.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.605133Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:16Z"} 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.605133_407.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.605133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.605133_407.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.605133_407.1 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:21.559+0000 7f19f5ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:21.560+0000 7f19f5ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:21Z"} 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.602798_406.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.602798Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:21Z"} 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.602798_406.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.602798Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.602798_406.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.602798_406.1 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:26.709+0000 7fb106e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:26.710+0000 7fb106e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:26Z"} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:26Z"} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:31.919+0000 7fd89362d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:31.921+0000 7fd89362d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:31Z"} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:31Z"} 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:37.171+0000 7f14daf30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:37.172+0000 7f14daf30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:37Z"} 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:37Z"} 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:42.305+0000 7fd486130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:42.306+0000 7fd486130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:42Z"} 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.617439_408.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.617439Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:42Z"} 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.617439_408.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.617439Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.617439_408.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.617439_408.1 2024-03-04T19:09:56.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:47.450+0000 7fe4ae022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:47.451+0000 7fe4ae022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:47Z"} 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:47Z"} 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:52.576+0000 7fb8ec62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:54:52.577+0000 7fb8ec62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:52Z"} 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:54:52Z"} 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709570988.692350_570.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-03-04T19:09:56.259 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1064, in test_multi_zone_redirect 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.260 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"K5KCDZD1BZNDFSI4","secret_key":"bzF8rd2s7PcgxB8QByY3P7dLjeiaYknm"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","notif_pool":"test-zone2.rgw.log:notif"} 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-03-04T19:09:56.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: Sending period to new master zone fc294220-5038-4f25-aeee-cac2c5838729 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","epoch":4,"predecessor_uuid":"11c84af7-2aba-4dd8-b74e-932e4b0eb09a","sync_status":[],"period_map":{"id":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","zones":[{"id":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","name":"test-zone2","endpoints":["http://smithi071.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","name":"test-zone3","endpoints":["http://smithi027.front.sepia.ceph.com:8001"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]},{"id":"fc294220-5038-4f25-aeee-cac2c5838729","name":"test-zone1","endpoints":["http://smithi027.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","sync_policy":{"groups":[]},"enabled_features":["resharding"]}],"short_zone_ids":[{"key":"a1482fc7-8de0-4f7d-8e05-b02ecec7d664","val":2952716642},{"key":"dce1012f-1cf8-44d9-b80b-aa1da8d03699","val":210978822},{"key":"fc294220-5038-4f25-aeee-cac2c5838729","val":3773364728}]},"master_zonegroup":"test-zonegroup","master_zone":"fc294220-5038-4f25-aeee-cac2c5838729","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"2fe35df2-135e-4762-bdce-6c2b1b264663","realm_epoch":2} 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone test-zone2 to false 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-48 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-48/ 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-48/ 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-48/ 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:55:57 GMT 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:/sbnvju-48/ 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:iu4mpwfyx0WX+B1UwAZAUdiB3yk= 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 16:55:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iu4mpwfyx0WX+B1UwAZAUdiB3yk=', 'Content-Length': '0'} 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d215f6a777946ae6-0065e5fd1d-5285-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 16:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-48/testredirect 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-48/testredirect 2024-03-04T19:09:56.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-48/testredirect 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 16:55:57 GMT 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:/sbnvju-48/testredirect 2024-03-04T19:09:56.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Wt+hCQK+q9Elh3WThm0/KuXfGJU= 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 16:55:57 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Wt+hCQK+q9Elh3WThm0/KuXfGJU='} 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adaf49366724cc99-0065e5fd1d-5285-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 16:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709571357.890340_643.1","last_update":"2024-03-04T16:55:57.890340Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:55:58.161+0000 7f0d8c135800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:55:58.162+0000 7f0d8c135800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:55:58Z"} 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:55:58Z"} 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:03.290+0000 7f3fa693c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:03.291+0000 7f3fa693c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:03Z"} 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:03Z"} 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:08.414+0000 7fe692c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:08.415+0000 7fe692c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:08Z"} 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:08Z"} 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:13.550+0000 7f94b832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:13.551+0000 7f94b832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:13Z"} 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:13Z"} 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:18.686+0000 7f7b68929800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:18.687+0000 7f7b68929800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:18Z"} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:18Z"} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:23.821+0000 7fde9c224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:23.822+0000 7fde9c224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:23Z"} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:23Z"} 2024-03-04T19:09:56.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:28.980+0000 7fb21e831800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:28.982+0000 7fb21e831800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:28Z"} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:28Z"} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi027.stdout: INFO: 2024-03-04T16:56:29.221+0000 7fc935f0d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:34.195+0000 7fd7ee328800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:34.195+0000 7fd7ee328800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:34Z"} 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:34Z"} 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:39.334+0000 7f88cc531800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:39.335+0000 7f88cc531800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:39Z"} 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:39Z"} 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:44.471+0000 7f4007844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:44.472+0000 7f4007844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:44Z"} 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:44Z"} 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:49.596+0000 7fb8ae88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:49.597+0000 7fb8ae88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:49Z"} 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:49Z"} 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:54.743+0000 7f2cbd11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:54.745+0000 7f2cbd11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:54Z"} 2024-03-04T19:09:56.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:54Z"} 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T16:56:59.203+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T16:56:59.204+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:59.865+0000 7f535e12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:56:59.866+0000 7f535e12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:59Z"} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:56:59Z"} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:05.005+0000 7febecf26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:05.006+0000 7febecf26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:05Z"} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:05Z"} 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:10.145+0000 7f6e62e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:10.145+0000 7f6e62e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:10Z"} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:10Z"} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:15.285+0000 7f24ede16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:15.285+0000 7f24ede16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:15Z"} 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:15Z"} 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:20.419+0000 7f8e51e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:20.420+0000 7f8e51e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:20Z"} 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:20Z"} 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:25.666+0000 7f5b5469d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:25.667+0000 7f5b5469d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:25Z"} 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:25Z"} 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:30.797+0000 7fec5ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:30.798+0000 7fec5ba2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:30Z"} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:30Z"} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:35.936+0000 7f907ae3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:35.937+0000 7f907ae3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:35Z"} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:35Z"} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:41.072+0000 7fd06c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:41.073+0000 7fd06c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:41Z"} 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:41Z"} 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:46.211+0000 7f270d214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:46.212+0000 7f270d214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:46Z"} 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:46Z"} 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:51.345+0000 7f22821a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:51.346+0000 7f22821a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:51Z"} 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:51Z"} 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:56.486+0000 7f3c7f667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:57:56.487+0000 7f3c7f667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:56Z"} 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:57:56Z"} 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:01.701+0000 7fca3bb30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:01.703+0000 7fca3bb30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:01Z"} 2024-03-04T19:09:56.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:01Z"} 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:06.842+0000 7fb758630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:06.843+0000 7fb758630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:06Z"} 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:06Z"} 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:11.980+0000 7f52b3044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:11.981+0000 7f52b3044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:11Z"} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:11Z"} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:17.134+0000 7f92d9739800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:17.135+0000 7f92d9739800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:17Z"} 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:17Z"} 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:22.269+0000 7f1de1413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:22.270+0000 7f1de1413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:22Z"} 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:22Z"} 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:27.489+0000 7fa21aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:27.493+0000 7fa21aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:27Z"} 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:27Z"} 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:32.647+0000 7f2bfa49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:32.648+0000 7f2bfa49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:32Z"} 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:32Z"} 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:37.787+0000 7f4488e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:37.787+0000 7f4488e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:37Z"} 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:37Z"} 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:42.937+0000 7fdd24711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:42.937+0000 7fdd24711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:42Z"} 2024-03-04T19:09:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:42Z"} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:48.059+0000 7f74d1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:48.060+0000 7f74d1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:48Z"} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:48Z"} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:53.187+0000 7f3a27a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:53.188+0000 7f3a27a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:53Z"} 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:53Z"} 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:58.308+0000 7fb2e014b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:58:58.309+0000 7fb2e014b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:58Z"} 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:58:58Z"} 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:03.449+0000 7f3730e38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:03.450+0000 7f3730e38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:03Z"} 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:03Z"} 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:08.593+0000 7fa2d2b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:08.594+0000 7fa2d2b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:08Z"} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.715026_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.715026Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:08Z"} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.715026_419.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.715026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.715026_419.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.715026_419.1 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:13.725+0000 7fc387d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:13.726+0000 7fc387d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:13Z"} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:13Z"} 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:18.924+0000 7f127c644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:18.926+0000 7f127c644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:18Z"} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:18Z"} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:24.057+0000 7f9fe61a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:24.058+0000 7f9fe61a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:24Z"} 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:24Z"} 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.799310_424.1 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:29.190+0000 7f033002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:29.191+0000 7f033002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:29Z"} 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:29Z"} 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:34.321+0000 7ffbcd89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:34.322+0000 7ffbcd89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:34Z"} 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.816633_428.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.816633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:34Z"} 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.816633_428.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.816633Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.816633_428.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.816633_428.1 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:39.441+0000 7f408f726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:39.442+0000 7f408f726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:39Z"} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:39Z"} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:44.567+0000 7f58ddc30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:44.568+0000 7f58ddc30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:44Z"} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:44Z"} 2024-03-04T19:09:56.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:49.705+0000 7f179e74b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:49.706+0000 7f179e74b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:49Z"} 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.799310_424.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.799310Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:49Z"} 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.799310_424.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.799310Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.799310_424.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.799310_424.1 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:54.936+0000 7fb06769d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T16:59:54.937+0000 7fb06769d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:54Z"} 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T16:59:54Z"} 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.738966_421.1 2024-03-04T19:09:56.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:00.061+0000 7f627c33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:00.062+0000 7f627c33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:00Z"} 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:00Z"} 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:05.199+0000 7fb197331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:05.199+0000 7fb197331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:05Z"} 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:05Z"} 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:10.320+0000 7ff873c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:10.321+0000 7ff873c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:10Z"} 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.815644_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.815644Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:10Z"} 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.815644_427.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.815644Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.815644_427.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.815644_427.1 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:15.457+0000 7f4fab920800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:15.458+0000 7f4fab920800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:15Z"} 2024-03-04T19:09:56.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:15Z"} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:20.681+0000 7fb161f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:20.684+0000 7fb161f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:20Z"} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:20Z"} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:25.872+0000 7f1c8052f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:25.873+0000 7f1c8052f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:25Z"} 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:25Z"} 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:31.017+0000 7fb9c8b3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:31.018+0000 7fb9c8b3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:31Z"} 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:31Z"} 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:36.143+0000 7fbe08da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:36.144+0000 7fbe08da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:36Z"} 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:36Z"} 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:41.268+0000 7f05f3644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:41.269+0000 7f05f3644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:41Z"} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:41Z"} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:46.416+0000 7f5f7062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:46.417+0000 7f5f7062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:46Z"} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:46Z"} 2024-03-04T19:09:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:51.655+0000 7f3d7892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:51.656+0000 7f3d7892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:51Z"} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:51Z"} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:56.812+0000 7f8d53511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:00:56.813+0000 7f8d53511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:56Z"} 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:00:56Z"} 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:01.954+0000 7f5373e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:01.955+0000 7f5373e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:01Z"} 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:01Z"} 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.316 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1093, in test_zonegroup_remove 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.317 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709571357.890340_643.1","last_update":"2024-03-04T16:55:57.890340Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:07.217+0000 7fccc8d2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:07.218+0000 7fccc8d2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:07Z"} 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:07Z"} 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:12.336+0000 7fe43180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:12.337+0000 7fe43180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:12Z"} 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:12Z"} 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:17.563+0000 7ff03a085800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:17.565+0000 7ff03a085800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:17Z"} 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:17Z"} 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:22.693+0000 7f3455733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:22.694+0000 7f3455733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:22Z"} 2024-03-04T19:09:56.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:22Z"} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:27.819+0000 7f677412b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:27.820+0000 7f677412b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:27Z"} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:27Z"} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:32.967+0000 7f97f9614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:32.968+0000 7f97f9614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:32Z"} 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:32Z"} 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:38.108+0000 7f2534b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:38.109+0000 7f2534b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:38Z"} 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:38Z"} 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:43.254+0000 7fa25e12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:43.255+0000 7fa25e12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:43Z"} 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:43Z"} 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:48.484+0000 7fc9c232b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:48.485+0000 7fc9c232b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:48Z"} 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:48Z"} 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi027.stdout: INFO: 2024-03-04T17:01:49.226+0000 7fc935f0d640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:53.622+0000 7faa93c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:53.623+0000 7faa93c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:53Z"} 2024-03-04T19:09:56.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:53Z"} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:58.756+0000 7fed1ca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:01:58.757+0000 7fed1ca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:58Z"} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:01:58Z"} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:03.888+0000 7f9568a3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:03.889+0000 7f9568a3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:03Z"} 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:03Z"} 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:09.019+0000 7f4ca2b39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:09.020+0000 7f4ca2b39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:09Z"} 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:09Z"} 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:14.172+0000 7f57d4ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:14.173+0000 7f57d4ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:14Z"} 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:14Z"} 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:02:19.262+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:19.321+0000 7fd8de33c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:19.322+0000 7fd8de33c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:19Z"} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:19Z"} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:24.527+0000 7f5f60111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:24.528+0000 7f5f60111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:24Z"} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:24Z"} 2024-03-04T19:09:56.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:29.675+0000 7fbc9b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:29.676+0000 7fbc9b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:29Z"} 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:29Z"} 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:34.814+0000 7f3e9e91f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:34.815+0000 7f3e9e91f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:34Z"} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:34Z"} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.333 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:02:39.206+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:39.952+0000 7f020c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:39.953+0000 7f020c733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:39Z"} 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:39Z"} 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:45.097+0000 7f8d33f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:45.098+0000 7f8d33f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:45Z"} 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:45Z"} 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:50.354+0000 7f0e42e15800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:50.357+0000 7f0e42e15800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:50Z"} 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:50Z"} 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:55.511+0000 7f5600424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:02:55.512+0000 7f5600424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:55Z"} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:02:55Z"} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:00.634+0000 7f6f48667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:00.635+0000 7f6f48667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:00Z"} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:00Z"} 2024-03-04T19:09:56.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:05.760+0000 7fb26de12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:05.761+0000 7fb26de12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:05Z"} 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:05Z"} 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:10.904+0000 7fe4a4e32800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:10.904+0000 7fe4a4e32800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:10Z"} 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:10Z"} 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:16.100+0000 7f2894e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:16.101+0000 7f2894e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:16Z"} 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:16Z"} 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:21.337+0000 7fb7b4613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:21.338+0000 7fb7b4613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:21Z"} 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:21Z"} 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:26.469+0000 7f0fd313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:26.470+0000 7f0fd313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:26Z"} 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:26Z"} 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:31.593+0000 7fead4414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:31.594+0000 7fead4414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:31Z"} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:31Z"} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:36.737+0000 7f62c482f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:36.738+0000 7f62c482f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:36Z"} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:36Z"} 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:41.874+0000 7f4861430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:41.875+0000 7f4861430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:41Z"} 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:41Z"} 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:47.004+0000 7fd76802e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:47.005+0000 7fd76802e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:47Z"} 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:47Z"} 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:52.154+0000 7f94959a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:52.155+0000 7f94959a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:52Z"} 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.795745_422.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.795745Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:52Z"} 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.795745_422.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.795745Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.795745_422.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.795745_422.1 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:57.287+0000 7f3566a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:03:57.288+0000 7f3566a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:57Z"} 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:03:57Z"} 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:02.489+0000 7ff4a972a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:02.491+0000 7ff4a972a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:02Z"} 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.813215_426.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.813215Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:02Z"} 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.813215_426.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.813215Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.813215_426.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.813215_426.1 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:07.613+0000 7f1a99a17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:07.613+0000 7f1a99a17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:07Z"} 2024-03-04T19:09:56.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:07Z"} 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:12.746+0000 7fbe1c539800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:12.747+0000 7fbe1c539800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:12Z"} 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.738966_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.738966Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:12Z"} 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.738966_421.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.738966Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.738966_421.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.738966_421.1 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:17.879+0000 7eff61d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:17.881+0000 7eff61d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:17Z"} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:17Z"} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:23.026+0000 7f1be2267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:23.027+0000 7f1be2267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:23Z"} 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:23Z"} 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:28.160+0000 7f3a97ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:28.161+0000 7f3a97ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:28Z"} 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.736351_420.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.736351Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:28Z"} 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.736351_420.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.736351Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.736351_420.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.736351_420.1 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:33.301+0000 7f8f88624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:33.301+0000 7f8f88624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:33Z"} 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:33Z"} 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:38.526+0000 7fc1f3e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:38.527+0000 7fc1f3e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:38Z"} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:38Z"} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:43.665+0000 7f5ad1d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:43.666+0000 7f5ad1d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:43Z"} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.647116_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.647116Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:43Z"} 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.647116_415.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.647116Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.647116_415.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.647116_415.1 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:48.808+0000 7f595480f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:48.809+0000 7f595480f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:48Z"} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:48Z"} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:53.937+0000 7f4818e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:53.938+0000 7f4818e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:53Z"} 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:53Z"} 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:59.073+0000 7f4fd980f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:04:59.074+0000 7f4fd980f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:59Z"} 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.659486_416.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.659486Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:04:59Z"} 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.659486_416.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.659486Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.659486_416.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.659486_416.1 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:04.209+0000 7fe63ea16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:04.210+0000 7fe63ea16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:04Z"} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:04Z"} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:09.345+0000 7f7e74c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:09.346+0000 7f7e74c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:09Z"} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:09Z"} 2024-03-04T19:09:56.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:14.571+0000 7fb748a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:14.576+0000 7fb748a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:14Z"} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:14Z"} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:19.732+0000 7f213e4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:19.733+0000 7f213e4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:19Z"} 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:19Z"} 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:24.856+0000 7f736c333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:24.857+0000 7f736c333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:24Z"} 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:24Z"} 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:29.987+0000 7f994622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:29.988+0000 7f994622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:29Z"} 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.623267_411.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.623267Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:29Z"} 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.623267_411.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.623267Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.623267_411.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.623267_411.1 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:35.136+0000 7f0e91333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:35.137+0000 7f0e91333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:35Z"} 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.819018_429.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.819018Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:35Z"} 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.819018_429.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.819018Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.819018_429.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.819018_429.1 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:40.324+0000 7f31b3d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:40.325+0000 7f31b3d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:40Z"} 2024-03-04T19:09:56.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.643873_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.643873Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:40Z"} 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.643873_413.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.643873Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.643873_413.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.643873_413.1 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:45.451+0000 7fcb43c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:45.452+0000 7fcb43c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:45Z"} 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:45Z"} 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:50.585+0000 7fa172f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:50.586+0000 7fa172f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:50Z"} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.711729_418.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.711729Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:50Z"} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.711729_418.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.711729Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.711729_418.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.711729_418.1 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:55.731+0000 7fa2f5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:05:55.731+0000 7fa2f5b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:55Z"} 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:05:55Z"} 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:01.001+0000 7f7faef1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:01.017+0000 7f7faef1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:01Z"} 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.641235_412.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.641235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:01Z"} 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.641235_412.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.641235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.641235_412.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.641235_412.1 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:06.476+0000 7f4e73816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:06.477+0000 7f4e73816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:06Z"} 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.644836_414.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.644836Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:06Z"} 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.644836_414.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.644836Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.644836_414.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.644836_414.1 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:11.610+0000 7f4c75430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:11.611+0000 7f4c75430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:11Z"} 2024-03-04T19:09:56.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.838183_430.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.838183Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:11Z"} 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.838183_430.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.838183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.838183_430.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571357.890340_643.1 target=1_1709570369.838183_430.1 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-03-04T19:09:56.369 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1176, in test_bucket_sync_disable 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-55 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-55/ 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-55/ 2024-03-04T19:09:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-55/ 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:06:20 GMT 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:/sbnvju-55/ 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:mndBUIiPUKa5Fwbir/0gIXzhTsk= 2024-03-04T19:09:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mndBUIiPUKa5Fwbir/0gIXzhTsk=', 'Content-Length': '0'} 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000636efaf50881980e-0065e5ff8c-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-56 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-56/ 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-56/ 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-56/ 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.372 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:06:20 GMT 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:/sbnvju-56/ 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:pR8DDo/vFdsie+jgeYv0jg5XP2I= 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pR8DDo/vFdsie+jgeYv0jg5XP2I=', 'Content-Length': '0'} 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000afe47f9ac55bf56a-0065e5ff8c-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-57 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-57/ 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-57/ 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-57/ 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:06:20 GMT 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:/sbnvju-57/ 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5naGR1meLVAGdOMauEe94k94r5Y= 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:06:20 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5naGR1meLVAGdOMauEe94k94r5Y=', 'Content-Length': '0'} 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b5b0cb2c7d1547eb-0065e5ff8c-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:06:20 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709571980.387781_747.1","last_update":"2024-03-04T17:06:20.387781Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:20.747+0000 7fd91a133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:20.748+0000 7fd91a133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:20Z"} 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:20Z"} 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:26.279+0000 7facb380f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:26.280+0000 7facb380f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:26Z"} 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:26Z"} 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:31.422+0000 7f2e72ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:31.423+0000 7f2e72ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:31Z"} 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:31Z"} 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:36.570+0000 7f766ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:36.571+0000 7f766ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:36Z"} 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:36Z"} 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:41.705+0000 7fe9d5616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:41.706+0000 7fe9d5616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:41Z"} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:41Z"} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570372.193667_464.1 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:46.855+0000 7f2f1ac17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:46.856+0000 7f2f1ac17800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:46Z"} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:46Z"} 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:51.985+0000 7f5ff9da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:51.986+0000 7f5ff9da2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:51Z"} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:51Z"} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:57.323+0000 7f7cf0213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:06:57.324+0000 7f7cf0213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:57Z"} 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:06:57Z"} 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:02.596+0000 7f71f6d2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:02.598+0000 7f71f6d2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:02Z"} 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:02Z"} 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:07.801+0000 7f7e7bb11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:07.802+0000 7f7e7bb11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:07Z"} 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:07Z"} 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:12.943+0000 7fca0a730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:12.944+0000 7fca0a730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:12Z"} 2024-03-04T19:09:56.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:12Z"} 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:18.085+0000 7f8679e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:18.085+0000 7f8679e18800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:18Z"} 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:18Z"} 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:23.218+0000 7fc26d82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:23.219+0000 7fc26d82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:23Z"} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:23Z"} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:28.456+0000 7fe97ce20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:28.457+0000 7fe97ce20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:28Z"} 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:28Z"} 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:33.589+0000 7ff8f7030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:33.590+0000 7ff8f7030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:33Z"} 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:33Z"} 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:38.710+0000 7f37db326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:38.711+0000 7f37db326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:38Z"} 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:38Z"} 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:43.836+0000 7f6240d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:43.837+0000 7f6240d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:43Z"} 2024-03-04T19:09:56.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:43Z"} 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:48.983+0000 7f427c53c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:48.984+0000 7f427c53c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:48Z"} 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:48Z"} 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:54.113+0000 7fbf14930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:54.114+0000 7fbf14930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:54Z"} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:54Z"} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:59.254+0000 7f1952c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:07:59.255+0000 7f1952c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:59Z"} 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:07:59Z"} 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:04.401+0000 7fe11b42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:04.401+0000 7fe11b42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:04Z"} 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:04Z"} 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:09.540+0000 7f8786812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:09.541+0000 7f8786812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:09Z"} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:09Z"} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:14.685+0000 7f50ffd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:14.686+0000 7f50ffd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:14Z"} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:14Z"} 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:08:18.959+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:08:18.997+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:19.826+0000 7f2d39a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:19.827+0000 7f2d39a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:19Z"} 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:19Z"} 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:24.983+0000 7ff7cb224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:24.984+0000 7ff7cb224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:24Z"} 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:24Z"} 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:30.133+0000 7f9c72e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:30.133+0000 7f9c72e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:30Z"} 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:30Z"} 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:35.281+0000 7ff7bf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:35.282+0000 7ff7bf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:35Z"} 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:35Z"} 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:40.438+0000 7f43e15a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:40.439+0000 7f43e15a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:40Z"} 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:40Z"} 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:45.625+0000 7f4decf11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:45.631+0000 7f4decf11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:45Z"} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:45Z"} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:50.917+0000 7f8135933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:50.918+0000 7f8135933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:50Z"} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:50Z"} 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:56.075+0000 7f1b93733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:08:56.076+0000 7f1b93733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:56Z"} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:08:56Z"} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:01.209+0000 7fbf7f139800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:01.210+0000 7fbf7f139800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:01Z"} 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:01Z"} 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:06.331+0000 7f3b8a911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:06.332+0000 7f3b8a911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:06Z"} 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:06Z"} 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:11.580+0000 7fc7af02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:11.581+0000 7fc7af02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:11Z"} 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:11Z"} 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:16.715+0000 7f030a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:16.716+0000 7f030a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:16Z"} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:16Z"} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:21.887+0000 7ff37d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:21.888+0000 7ff37d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:21Z"} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:21Z"} 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:27.015+0000 7f872e20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:27.016+0000 7f872e20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:27Z"} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:27Z"} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:32.329+0000 7f137c54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:32.330+0000 7f137c54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:32Z"} 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:32Z"} 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:37.477+0000 7f085bd30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:37.478+0000 7f085bd30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:37Z"} 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:37Z"} 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:42.617+0000 7f6620d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:42.618+0000 7f6620d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:42Z"} 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:42Z"} 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:47.746+0000 7fcd5c133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:47.747+0000 7fcd5c133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:47Z"} 2024-03-04T19:09:56.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:47Z"} 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:52.875+0000 7f4cd9e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:52.876+0000 7f4cd9e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:52Z"} 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:52Z"} 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:58.010+0000 7fd01e616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:09:58.011+0000 7fd01e616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:58Z"} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:09:58Z"} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:03.165+0000 7f010fc67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:03.166+0000 7f010fc67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:03Z"} 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:03Z"} 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:08.301+0000 7f00fa133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:08.302+0000 7f00fa133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:08Z"} 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:08Z"} 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:13.429+0000 7fc8d6e2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:13.430+0000 7fc8d6e2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:13Z"} 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:13Z"} 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:18.571+0000 7f95e4e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:18.591+0000 7f95e4e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:18Z"} 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:18Z"} 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:23.909+0000 7f831988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:23.910+0000 7f831988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:23Z"} 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:23Z"} 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:29.040+0000 7f4a6be11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:29.041+0000 7f4a6be11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:29Z"} 2024-03-04T19:09:56.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:29Z"} 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:34.175+0000 7f99755a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:34.176+0000 7f99755a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:34Z"} 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:34Z"} 2024-03-04T19:09:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:39.302+0000 7f432308c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:39.302+0000 7f432308c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:39Z"} 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:39Z"} 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:44.527+0000 7fd653610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:44.528+0000 7fd653610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:44Z"} 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:44Z"} 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:49.664+0000 7f113c530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:49.665+0000 7f113c530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:49Z"} 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:49Z"} 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:54.810+0000 7f556f73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:54.811+0000 7f556f73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:54Z"} 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:54Z"} 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:59.944+0000 7f4fb7a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:10:59.945+0000 7f4fb7a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:59Z"} 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:10:59Z"} 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:05.083+0000 7ff3af2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:05.084+0000 7ff3af2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:05Z"} 2024-03-04T19:09:56.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:05Z"} 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:10.224+0000 7f9e14e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:10.225+0000 7f9e14e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:10Z"} 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:10Z"} 2024-03-04T19:09:56.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:15.364+0000 7fbd7a20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:15.365+0000 7fbd7a20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:15Z"} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:15Z"} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:20.496+0000 7f50732c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:20.497+0000 7f50732c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:20Z"} 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:20Z"} 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:25.639+0000 7f5008011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:25.640+0000 7f5008011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:25Z"} 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:25Z"} 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709571980.387781_747.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709571980.387781_747.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.425 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1200, in test_bucket_sync_enable_right_after_disable 2024-03-04T19:09:56.426 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-58 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/ 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/ 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/ 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:/sbnvju-58/ 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:MxGvxYwl/d+ex3qbaQvROa3nuhM= 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:MxGvxYwl/d+ex3qbaQvROa3nuhM=', 'Content-Length': '0'} 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fd5297ecfda2077c-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-59 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/ 2024-03-04T19:09:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/ 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/ 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:/sbnvju-59/ 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:iDhR2VwJaiWslRAUgAx4VZuQSoc= 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iDhR2VwJaiWslRAUgAx4VZuQSoc=', 'Content-Length': '0'} 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f1392431780d9460-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-60 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/ 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/ 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/ 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.433 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:/sbnvju-60/ 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Bc/Bqz2JExG0iAANJjSQt147Pyg= 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Bc/Bqz2JExG0iAANJjSQt147Pyg=', 'Content-Length': '0'} 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000846a5fe8b5d8134-0065e600c2-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/ 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/ 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/ 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:/sbnvju-58/ 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091ace23e93c1c288-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/obj1 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/obj1 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/obj1 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:/sbnvju-58/obj1 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:m/tUU+merGqraCwlOLvyonrIsYo= 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:m/tUU+merGqraCwlOLvyonrIsYo='} 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058a331f2a1bd2ac2-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/ 2024-03-04T19:09:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/ 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/ 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:/sbnvju-58/ 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.437 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000792a0a1509516d98-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/obj2 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/obj2 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/obj2 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.438 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:/sbnvju-58/obj2 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:NYgGjUIDTKi2Sr8U2NQaojJSrTI= 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:NYgGjUIDTKi2Sr8U2NQaojJSrTI='} 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cba7c552b87ba2bb-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/ 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/ 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/ 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:/sbnvju-58/ 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002358189e4c109f38-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/obj3 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/obj3 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/obj3 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:/sbnvju-58/obj3 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:s4tddDViq/IOPJofDBOHfdMT05g= 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s4tddDViq/IOPJofDBOHfdMT05g='} 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008772bb9a5d2c5efd-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/ 2024-03-04T19:09:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/ 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/ 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:/sbnvju-58/ 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.442 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY= 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IU2b71p6S6c6i+bH6wTGOtArnaY=', 'Content-Length': '0'} 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f6ea4651459d63-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-58/obj4 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-58/obj4 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-58/obj4 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.443 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:/sbnvju-58/obj4 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2R3e9VN6f2eYp9g+pxK30DzWe8o= 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2R3e9VN6f2eYp9g+pxK30DzWe8o='} 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c68d8dfed497a337-0065e600c2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/ 2024-03-04T19:09:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/ 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/ 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.445 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:/sbnvju-59/ 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008095ff1cece9a32c-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/obj1 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/obj1 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/obj1 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:/sbnvju-59/obj1 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:QoVRB8IhsWUJ7G2xA7i/Z9k5Y/Y= 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QoVRB8IhsWUJ7G2xA7i/Z9k5Y/Y='} 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000768e9440bb15dfa4-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/ 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/ 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/ 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:/sbnvju-59/ 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000571f24dc1ede9b8d-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/obj2 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/obj2 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/obj2 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.449 INFO:tasks.rgw_multisite_tests:/sbnvju-59/obj2 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:sXTH6NIbYnmpKhF8rwoYA7vFJHk= 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:sXTH6NIbYnmpKhF8rwoYA7vFJHk='} 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2bb71f9f5849460-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/ 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/ 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/ 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:/sbnvju-59/ 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b6469d48d2f1524-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/obj3 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/obj3 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/obj3 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:/sbnvju-59/obj3 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:WYxUANy1Z6NAoPrzvbUk1/y6Aeo= 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:WYxUANy1Z6NAoPrzvbUk1/y6Aeo='} 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c91f5c552f56a22-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/ 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/ 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/ 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:/sbnvju-59/ 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY= 2024-03-04T19:09:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:V6+xpqfrYDCZceGGSHNNfyl63mY=', 'Content-Length': '0'} 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b15916e85a7c5bcf-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-59/obj4 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-59/obj4 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-59/obj4 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.454 INFO:tasks.rgw_multisite_tests:/sbnvju-59/obj4 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:+izsiITcpqkbnvB1bzHJuoHX4Yk= 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+izsiITcpqkbnvB1bzHJuoHX4Yk='} 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6fb78e393b7615-0065e600c2-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:30 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/ 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/ 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/ 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.455 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:30 GMT 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:/sbnvju-60/ 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:t6VGvEsxsE+0aHfCU6aCrQkGFuI= 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:30 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:t6VGvEsxsE+0aHfCU6aCrQkGFuI=', 'Content-Length': '0'} 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073df105823d98088-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/obj1 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/obj1 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/obj1 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:/sbnvju-60/obj1 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TvN9T2rC5Lp1aXLJhNnmibG5bmI= 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TvN9T2rC5Lp1aXLJhNnmibG5bmI='} 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b966b99d2c6b47-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/ 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/ 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/ 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:/sbnvju-60/ 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dbb5830764abcda-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/obj2 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/obj2 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/obj2 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:/sbnvju-60/obj2 2024-03-04T19:09:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:aokVgM9o3MkCPWDZvTfJNlNV0zM= 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:aokVgM9o3MkCPWDZvTfJNlNV0zM='} 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030e0f65a3f183b18-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/ 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/ 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/ 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.460 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:/sbnvju-60/ 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693353b7acaecdb1-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/obj3 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/obj3 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/obj3 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:/sbnvju-60/obj3 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:AIYYSPbLNTgTXlBJoMzi9tdEAL4= 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:AIYYSPbLNTgTXlBJoMzi9tdEAL4='} 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a858cf4c1bcdd25-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/ 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/ 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/ 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:/sbnvju-60/ 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg= 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Esb0vCuEYae3SAQ88O66L1goTxg=', 'Content-Length': '0'} 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d3e0fc9963ad62-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-60/obj4 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-60/obj4 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-60/obj4 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:11:31 GMT 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:/sbnvju-60/obj4 2024-03-04T19:09:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:s0HjZmJtTgygGQ82Kgdb0fseULo= 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:11:31 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:s0HjZmJtTgygGQ82Kgdb0fseULo='} 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec00a4b90c7e0cb0-0065e600c3-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:11:31 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709572290.757934_773.1","last_update":"2024-03-04T17:11:30.757934Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:31.309+0000 7f52be309800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:31.310+0000 7f52be309800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:31Z"} 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:31Z"} 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.954325_443.1 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:36.514+0000 7f765d430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:36.515+0000 7f765d430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:36Z"} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:36Z"} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:41.654+0000 7f58eec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:41.655+0000 7f58eec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:41Z"} 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:41Z"} 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:46.796+0000 7f478a32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:46.797+0000 7f478a32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:46Z"} 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:46Z"} 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:51.946+0000 7f1ab5faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:51.947+0000 7f1ab5faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:51Z"} 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:51Z"} 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:57.077+0000 7f1405a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:11:57.077+0000 7f1405a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:57Z"} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:11:57Z"} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:02.223+0000 7f5cc3f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:02.224+0000 7f5cc3f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:02Z"} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:02Z"} 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:07.368+0000 7fafd2624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:07.369+0000 7fafd2624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:07Z"} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:07Z"} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:12.512+0000 7f1ccb12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:12.513+0000 7f1ccb12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:12Z"} 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:12Z"} 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:17.664+0000 7ff07c018800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:17.665+0000 7ff07c018800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:17Z"} 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:17Z"} 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:22.819+0000 7f2ad7e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:22.820+0000 7f2ad7e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:22Z"} 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:22Z"} 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:27.957+0000 7f5275939800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:27.958+0000 7f5275939800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:27Z"} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:27Z"} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:33.081+0000 7fe79bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:33.082+0000 7fe79bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:33Z"} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:33Z"} 2024-03-04T19:09:56.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:38.231+0000 7f7209015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:38.232+0000 7f7209015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:38Z"} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:38Z"} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:43.378+0000 7f4ec7416800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:43.379+0000 7f4ec7416800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:43Z"} 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:43Z"} 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:48.519+0000 7f3c86e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:48.520+0000 7f3c86e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:48Z"} 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:48Z"} 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:53.672+0000 7f7ef1444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:53.673+0000 7f7ef1444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:53Z"} 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:53Z"} 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:58.817+0000 7f0128a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:12:58.818+0000 7f0128a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:58Z"} 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:12:58Z"} 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:04.053+0000 7ffa6b239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:04.054+0000 7ffa6b239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:04Z"} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:04Z"} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:09.287+0000 7fec9034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:09.288+0000 7fec9034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:09Z"} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:09Z"} 2024-03-04T19:09:56.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:14.430+0000 7f9d32639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:14.431+0000 7f9d32639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:14Z"} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:14Z"} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:13:19.013+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:19.605+0000 7fcc3ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:19.606+0000 7fcc3ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:19Z"} 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:19Z"} 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:24.727+0000 7ff96a63a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:24.728+0000 7ff96a63a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:24Z"} 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:24Z"} 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.954325_443.1 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:29.856+0000 7f69734c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:29.857+0000 7f69734c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:29Z"} 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:29Z"} 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:35.013+0000 7f6a9cc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:35.014+0000 7f6a9cc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:35Z"} 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:35Z"} 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:40.148+0000 7f4324133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:40.149+0000 7f4324133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.196251_465.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.196251Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:40Z"} 2024-03-04T19:09:56.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.196251_465.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.196251Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:40Z"} 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.196251_465.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.196251Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.196251_465.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.196251_465.1 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:45.289+0000 7f18ebec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:45.290+0000 7f18ebec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:45Z"} 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:45Z"} 2024-03-04T19:09:56.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:50.415+0000 7fbb5b244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:50.416+0000 7fbb5b244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:50Z"} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:50Z"} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:55.548+0000 7f7e8614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:13:55.549+0000 7f7e8614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:55Z"} 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:13:55Z"} 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:13:59.002+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:00.668+0000 7f6f9fe12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:00.669+0000 7f6f9fe12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:00Z"} 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:00Z"} 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:05.822+0000 7fd7171af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:05.823+0000 7fd7171af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:05Z"} 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:05Z"} 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:10.958+0000 7fd17a329800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:10.959+0000 7fd17a329800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:10Z"} 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:10Z"} 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:16.094+0000 7f5326f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:16.095+0000 7f5326f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:16Z"} 2024-03-04T19:09:56.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:16Z"} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:21.225+0000 7f1fdea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:21.226+0000 7f1fdea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:21Z"} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:21Z"} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:26.363+0000 7fd63028c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:26.364+0000 7fd63028c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:26Z"} 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:26Z"} 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:31.514+0000 7f2be4c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:31.515+0000 7f2be4c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:31Z"} 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:31Z"} 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:36.654+0000 7f5843e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:36.655+0000 7f5843e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:36Z"} 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:36Z"} 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:41.797+0000 7fa9dff29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:41.798+0000 7fa9dff29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:41Z"} 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:41Z"} 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:46.952+0000 7f8a5a01e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:46.953+0000 7f8a5a01e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:46Z"} 2024-03-04T19:09:56.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:46Z"} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:52.101+0000 7f72b4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:52.102+0000 7f72b4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:52Z"} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:52Z"} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:57.243+0000 7fa5c4d20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:14:57.244+0000 7fa5c4d20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:57Z"} 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:14:57Z"} 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:02.371+0000 7f1abda38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:02.372+0000 7f1abda38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:02Z"} 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:02Z"} 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:07.505+0000 7f0f6402e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:07.505+0000 7f0f6402e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:07Z"} 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:07Z"} 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:12.660+0000 7f8be1d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:12.661+0000 7f8be1d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:12Z"} 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:12Z"} 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:17.801+0000 7f5f04d1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:17.802+0000 7f5f04d1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:17Z"} 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:17Z"} 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:22.944+0000 7f5028931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:22.945+0000 7f5028931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:22Z"} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:22Z"} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:28.088+0000 7f4c75333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:28.089+0000 7f4c75333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:28Z"} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:28Z"} 2024-03-04T19:09:56.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:33.233+0000 7f8ba2137800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:33.234+0000 7f8ba2137800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:33Z"} 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:33Z"} 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:38.387+0000 7fa1a4731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:38.387+0000 7fa1a4731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:38Z"} 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:38Z"} 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:43.535+0000 7f043fc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:43.536+0000 7f043fc11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:43Z"} 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:43Z"} 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.218063_467.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.218063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.218063_467.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.218063_467.1 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:48.663+0000 7f3d6ebaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:48.664+0000 7f3d6ebaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:48Z"} 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:48Z"} 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.155110_458.1 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:53.799+0000 7f103ea3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:53.800+0000 7f103ea3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:53Z"} 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:53Z"} 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:58.961+0000 7fdf34e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:15:58.962+0000 7fdf34e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:58Z"} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:15:58Z"} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:04.118+0000 7f4d7de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:04.119+0000 7f4d7de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:04Z"} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:04Z"} 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:09.270+0000 7fae09816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:09.271+0000 7fae09816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:09Z"} 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:09Z"} 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:14.415+0000 7f9e20d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:14.416+0000 7f9e20d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:14Z"} 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:14Z"} 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570372.189913_462.1 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:19.554+0000 7fdccc630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:19.555+0000 7fdccc630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:19Z"} 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:19Z"} 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:24.688+0000 7fa7a2815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:24.689+0000 7fa7a2815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:24Z"} 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:24Z"} 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:29.929+0000 7f872549d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:29.934+0000 7f872549d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:29Z"} 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:29Z"} 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:35.123+0000 7f6397431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:35.124+0000 7f6397431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:35Z"} 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:35Z"} 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572290.757934_773.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572290.757934_773.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-03-04T19:09:56.517 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1235, in test_bucket_sync_disable_enable 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.518 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-61 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/ 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/ 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/ 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.519 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:/sbnvju-61/ 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:hhLCTjC1f1fDJ+oQyyA+u/A2C8g= 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:hhLCTjC1f1fDJ+oQyyA+u/A2C8g=', 'Content-Length': '0'} 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f18900e06d10ec48-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-62 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/ 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/ 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/ 2024-03-04T19:09:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:/sbnvju-62/ 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ijyKp7E2cgZL02krlImGwo6kTq8= 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ijyKp7E2cgZL02krlImGwo6kTq8=', 'Content-Length': '0'} 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001f53de1578e3947f-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-63 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/ 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/ 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/ 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.522 INFO:tasks.rgw_multisite_tests:/sbnvju-63/ 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:mnKjxPMU/nv5l3gF3EHNRddYBkk= 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:mnKjxPMU/nv5l3gF3EHNRddYBkk=', 'Content-Length': '0'} 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001ab8ed325ad5c528-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/ 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/ 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/ 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.523 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:/sbnvju-61/ 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002357bbda9c7dc435-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/obj1 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/obj1 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/obj1 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:/sbnvju-61/obj1 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kJzY2m2p4eki0fsHvNR5MQptQEU= 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kJzY2m2p4eki0fsHvNR5MQptQEU='} 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3629b655009a7a0-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/ 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/ 2024-03-04T19:09:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/ 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:/sbnvju-61/ 2024-03-04T19:09:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af5db6cba5ba8d1-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/obj2 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/obj2 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/obj2 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.527 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:/sbnvju-61/obj2 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:4U59cYN9pCmfYQ/obuOecZXSHsM= 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:4U59cYN9pCmfYQ/obuOecZXSHsM='} 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cb7891d58386c31-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/ 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/ 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/ 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:/sbnvju-61/ 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ccb959cc06a5ea2-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/obj3 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/obj3 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/obj3 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:/sbnvju-61/obj3 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5585jV42lQDJzf1fXAjE9urUYhY= 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5585jV42lQDJzf1fXAjE9urUYhY='} 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1a3129babacad07-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/ 2024-03-04T19:09:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/ 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/ 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:/sbnvju-61/ 2024-03-04T19:09:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo= 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jS7qmwldRj14IsvuPTLjSJ94PYo=', 'Content-Length': '0'} 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a91a1c793e32d40-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-61/obj4 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-61/obj4 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-61/obj4 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.532 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:/sbnvju-61/obj4 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:qPSEBSMHHlskUq+S4htfDRajIyM= 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:qPSEBSMHHlskUq+S4htfDRajIyM='} 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039250795f796b145-0065e601f8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/ 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/ 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/ 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:/sbnvju-62/ 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b51998707529d2-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/obj1 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/obj1 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/obj1 2024-03-04T19:09:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:/sbnvju-62/obj1 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:6oCTTk/eoIQlIZkGVdsEMz5KeA8= 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:6oCTTk/eoIQlIZkGVdsEMz5KeA8='} 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000332b0016cb7a836e-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/ 2024-03-04T19:09:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/ 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/ 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.536 INFO:tasks.rgw_multisite_tests:/sbnvju-62/ 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c3300d8002e8e13-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/obj2 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/obj2 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/obj2 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.537 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:/sbnvju-62/obj2 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:EzD7ujZ5Dk9Wb1iNj+02k/xP61c= 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:EzD7ujZ5Dk9Wb1iNj+02k/xP61c='} 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e09a082fbee0dc-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/ 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/ 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/ 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:/sbnvju-62/ 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dd97678887da140-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/obj3 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/obj3 2024-03-04T19:09:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/obj3 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:/sbnvju-62/obj3 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:nCprvWwXht/gV2RHA2N798FWbO8= 2024-03-04T19:09:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:nCprvWwXht/gV2RHA2N798FWbO8='} 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c14309132b08af37-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/ 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/ 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/ 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.541 INFO:tasks.rgw_multisite_tests:/sbnvju-62/ 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0= 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:kpB/JakplwCO1+k6QwgWxUWJ5e0=', 'Content-Length': '0'} 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7ef0f5b5ff9c157-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-62/obj4 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-62/obj4 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-62/obj4 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.542 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:/sbnvju-62/obj4 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:CloIwqoLxajq1ax6oBnWTrTlHdg= 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CloIwqoLxajq1ax6oBnWTrTlHdg='} 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061989dbd4ed6fbfb-0065e601f8-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/ 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/ 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/ 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:/sbnvju-63/ 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044ca7a0efefe427e-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/obj1 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/obj1 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/obj1 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:/sbnvju-63/obj1 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Y/oOq02/GGm8zsKqyxy5LHQ9R58= 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Y/oOq02/GGm8zsKqyxy5LHQ9R58='} 2024-03-04T19:09:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c9bd1a0615215aa-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/ 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/ 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/ 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.546 INFO:tasks.rgw_multisite_tests:/sbnvju-63/ 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3fd20728ec9c500-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/obj2 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/obj2 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/obj2 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.547 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:/sbnvju-63/obj2 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:iSVlsue2voyFYFrBy4VtMgALB4Q= 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:iSVlsue2voyFYFrBy4VtMgALB4Q='} 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfbee6d0dd4f2168-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/ 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/ 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/ 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:/sbnvju-63/ 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbb4f4ccdb977e94-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/obj3 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/obj3 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/obj3 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:/sbnvju-63/obj3 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:OP3i0fxF0SoHXTaBQH+tJq5pJS0= 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:OP3i0fxF0SoHXTaBQH+tJq5pJS0='} 2024-03-04T19:09:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094f664a2a4baa2cf-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/ 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/ 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/ 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.551 INFO:tasks.rgw_multisite_tests:/sbnvju-63/ 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8= 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:rBvtBO+OvW60qh/NefESXqnPKj8=', 'Content-Length': '0'} 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e52f86c991ba534-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-63/obj4 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-63/obj4 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-63/obj4 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:16:40 GMT 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:/sbnvju-63/obj4 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:oKh9wQ0r0F8KTD+ANl8yR3q2DM8= 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:16:40 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:oKh9wQ0r0F8KTD+ANl8yR3q2DM8='} 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0f914daddaf7f25-0065e601f8-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:16:40 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709572600.239122_819.1","last_update":"2024-03-04T17:16:40.239122Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:40.752+0000 7f4837c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:40.753+0000 7f4837c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:40Z"} 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:40Z"} 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:45.884+0000 7fec0b816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:45.885+0000 7fec0b816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:45Z"} 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:45Z"} 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:51.029+0000 7f35ddb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:51.030+0000 7f35ddb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:51Z"} 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:51Z"} 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:56.167+0000 7f7fce813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:16:56.168+0000 7f7fce813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:56Z"} 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:16:56Z"} 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:01.300+0000 7f4cd420f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:01.301+0000 7f4cd420f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:01Z"} 2024-03-04T19:09:56.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:01Z"} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:06.553+0000 7f43c2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:06.554+0000 7f43c2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:06Z"} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:06Z"} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:11.680+0000 7fbf1bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:11.681+0000 7fbf1bb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:11Z"} 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:11Z"} 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:16.819+0000 7f644472e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:16.820+0000 7f644472e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:16Z"} 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:16Z"} 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:21.961+0000 7f00ab630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:21.961+0000 7f00ab630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:21Z"} 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:21Z"} 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:27.096+0000 7f3ca8b35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:27.096+0000 7f3ca8b35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:27Z"} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:27Z"} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:32.238+0000 7f8b42931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:32.239+0000 7f8b42931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:32Z"} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:32Z"} 2024-03-04T19:09:56.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:37.379+0000 7fdd7c53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:37.380+0000 7fdd7c53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:37Z"} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:37Z"} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:42.520+0000 7fa9d0613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:42.521+0000 7fa9d0613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:42Z"} 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:42Z"} 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:47.668+0000 7fc39214b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:47.668+0000 7fc39214b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:47Z"} 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:47Z"} 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:52.815+0000 7f76f274b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:52.815+0000 7f76f274b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:52Z"} 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:52Z"} 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:57.942+0000 7f2a5a127800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:17:57.943+0000 7f2a5a127800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:57Z"} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:17:57Z"} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:03.077+0000 7f2fa1333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:03.078+0000 7f2fa1333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:03Z"} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:03Z"} 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:08.229+0000 7f84b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:08.229+0000 7f84b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:08Z"} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:08Z"} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:13.358+0000 7f324742d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:13.359+0000 7f324742d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:13Z"} 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:13Z"} 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:18.505+0000 7fa046424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:18.506+0000 7fa046424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:18Z"} 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:18Z"} 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:18:23.426+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:23.663+0000 7f45a2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:23.663+0000 7f45a2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:23Z"} 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:23Z"} 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:28.952+0000 7faf6bc39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:28.956+0000 7faf6bc39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:28Z"} 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:28Z"} 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.954325_443.1 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:34.160+0000 7f282c731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:34.161+0000 7f282c731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:34Z"} 2024-03-04T19:09:56.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:34Z"} 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:39.317+0000 7fc4a932e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:39.318+0000 7fc4a932e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:39Z"} 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:39Z"} 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:44.450+0000 7f660871e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:44.451+0000 7f660871e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:44Z"} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:44Z"} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:49.587+0000 7fe24573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:49.588+0000 7fe24573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:49Z"} 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:49Z"} 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:54.712+0000 7f8cf7e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:54.713+0000 7f8cf7e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:54Z"} 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:54Z"} 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:59.843+0000 7f15c812f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:18:59.844+0000 7f15c812f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:59Z"} 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:18:59Z"} 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:04.975+0000 7f7b552c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:04.976+0000 7f7b552c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:04Z"} 2024-03-04T19:09:56.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:04Z"} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:10.106+0000 7f9691c15800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:10.107+0000 7f9691c15800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:10Z"} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:10Z"} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:15.255+0000 7f9a153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:15.256+0000 7f9a153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:15Z"} 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:15Z"} 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:20.378+0000 7fb17d10d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:20.379+0000 7fb17d10d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:20Z"} 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:20Z"} 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:19:21.294+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:25.518+0000 7f1e66e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:25.519+0000 7f1e66e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:25Z"} 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:25Z"} 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:30.662+0000 7f42f3c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:30.663+0000 7f42f3c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:30Z"} 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:30Z"} 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:35.806+0000 7fa25e32d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:35.807+0000 7fa25e32d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:35Z"} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:35Z"} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:40.941+0000 7f769c667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:40.942+0000 7f769c667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:40Z"} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:40Z"} 2024-03-04T19:09:56.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:46.083+0000 7f68c813b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:46.084+0000 7f68c813b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:46Z"} 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:46Z"} 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:51.226+0000 7fec89533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:51.227+0000 7fec89533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:51Z"} 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:51Z"} 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:56.370+0000 7fafce89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:19:56.371+0000 7fafce89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:56Z"} 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:19:56Z"} 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:01.527+0000 7fa79cf2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:01.528+0000 7fa79cf2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:01Z"} 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:01Z"} 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:06.672+0000 7f4dda72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:06.673+0000 7f4dda72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:06Z"} 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:06Z"} 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:11.828+0000 7fad1249d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:11.829+0000 7fad1249d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:11Z"} 2024-03-04T19:09:56.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:11Z"} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:17.116+0000 7f7549639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:17.117+0000 7f7549639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:17Z"} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:17Z"} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:22.301+0000 7f77b7f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:22.302+0000 7f77b7f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:22Z"} 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:22Z"} 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:27.458+0000 7eff13044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:27.459+0000 7eff13044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:27Z"} 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:27Z"} 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:32.597+0000 7f7c5d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:32.598+0000 7f7c5d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:32Z"} 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.870879_436.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.870879Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:32Z"} 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.870879_436.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.870879Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.870879_436.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.870879_436.1 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:37.741+0000 7ff029c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:37.741+0000 7ff029c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:37Z"} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:37Z"} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:42.890+0000 7f04b412f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:42.891+0000 7f04b412f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:42Z"} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.927093_438.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.927093Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:42Z"} 2024-03-04T19:09:56.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.927093_438.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.927093Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.927093_438.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.927093_438.1 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:48.025+0000 7f9f4f7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:48.026+0000 7f9f4f7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:48Z"} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:48Z"} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:53.164+0000 7f7c6714b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:53.165+0000 7f7c6714b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:53Z"} 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.954325_443.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.954325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:53Z"} 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.954325_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.954325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.954325_443.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.954325_443.1 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:58.298+0000 7fa671242800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:20:58.299+0000 7fa671242800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:58Z"} 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:20:58Z"} 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:03.427+0000 7fad1d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:03.428+0000 7fad1d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:03Z"} 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:03Z"} 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:08.551+0000 7f901ba30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:08.552+0000 7f901ba30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:08Z"} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:08Z"} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:13.679+0000 7fcfec331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:13.680+0000 7fcfec331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:13Z"} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:13Z"} 2024-03-04T19:09:56.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:18.809+0000 7fd5f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:18.810+0000 7fd5f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:18Z"} 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:18Z"} 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:23.951+0000 7fe56a48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:23.952+0000 7fe56a48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:23Z"} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:23Z"} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.078213_520.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.078213Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.078213_520.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570680.078213_520.1 2024-03-04T19:09:56.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:29.093+0000 7f2dd95af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:29.094+0000 7f2dd95af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:29Z"} 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:29Z"} 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:34.242+0000 7fa28a824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:34.243+0000 7fa28a824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:34Z"} 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.950988_441.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.950988Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:34Z"} 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.950988_441.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.950988Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.950988_441.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.950988_441.1 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:39.383+0000 7fdb3f016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:39.384+0000 7fdb3f016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:39Z"} 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:39Z"} 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:44.629+0000 7f980cd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:44.632+0000 7f980cd3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:44Z"} 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.948368_440.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.948368Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:44Z"} 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.948368_440.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.948368Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572600.239122_819.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.948368_440.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572600.239122_819.1 target=1_1709570369.948368_440.1 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-03-04T19:09:56.604 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1279, in test_multipart_object_sync 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-64 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/ 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/ 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/ 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:49 GMT 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:/sbnvju-64/ 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:CEYLftg7jED1JErGZIwefD3VUYI= 2024-03-04T19:09:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:CEYLftg7jED1JErGZIwefD3VUYI=', 'Content-Length': '0'} 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003166221d0afb6959-0065e6032d-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=sbnvju-65 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-65/ 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-65/ 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-65/ 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.607 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:49 GMT 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:/sbnvju-65/ 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:IBomHfCt0HBbvYoy0fyRkqJePHs= 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:IBomHfCt0HBbvYoy0fyRkqJePHs=', 'Content-Length': '0'} 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c4edc14516a3876e-0065e6032d-7072-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=sbnvju-66 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-66/ 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-66/ 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-66/ 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:49 GMT 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:/sbnvju-66/ 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:ped9zcZDxZWxZyOE8O/9TY/8T7k= 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:ped9zcZDxZWxZyOE8O/9TY/8T7k=', 'Content-Length': '0'} 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e32b78d169679a90-0065e6032d-5358-test-zone3'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:21:49 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploads 2024-03-04T19:09:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploads 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploads 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:49 GMT 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?uploads 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.610 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:c6HudOOCO44uKcFMnsJnFWNNeIE= 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:49 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:c6HudOOCO44uKcFMnsJnFWNNeIE=', 'Content-Length': '0'} 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000eb315d133ee963df-0065e6032d-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'sbnvju-64MULTIPART2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2' 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=1 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.611 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?partNumber=1&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:prVxvh/BP37jaEmBaXL1oq/rM5Y= 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:prVxvh/BP37jaEmBaXL1oq/rM5Y='} 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d538b32d26173d34-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=2 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?partNumber=2&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:jAk4IGBYRSaNeVA+n1rtn8TTn50= 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:jAk4IGBYRSaNeVA+n1rtn8TTn50='} 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000713836097de8379f-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=3 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?partNumber=3&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:pxvwniKCLg8XEJdKOZET1gDg3po= 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:pxvwniKCLg8XEJdKOZET1gDg3po='} 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ceb167fd0781eaa2-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2&partNumber=4 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?partNumber=4&uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Uo838De1lPXbeVWYHn2wKs3VrYA= 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Uo838De1lPXbeVWYHn2wKs3VrYA='} 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c5727c08a89c055-0065e6032f-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:GET 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Tfz00BaRNCM8UDJUgLEMaHG+tsY= 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Tfz00BaRNCM8UDJUgLEMaHG+tsY=', 'Content-Length': '0'} 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083439ded0b5e3fb1-0065e6032f-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 1 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 3 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2024-03-04T19:09:56.618 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: 4 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:text/xml 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:21:51 GMT 2024-03-04T19:09:56.619 INFO:tasks.rgw_multisite_tests:/sbnvju-64/MULTIPART?uploadId=2~hqNfjhxIm5wcHlYRNbJHm9eOxGHmTB2 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:T/Eb3wJ+IjDfHryw8o4t+2z5/9A= 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:21:51 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:T/Eb3wJ+IjDfHryw8o4t+2z5/9A=', 'Content-Length': '460'} 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e2106474bc286131-0065e6032f-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Mon, 04 Mar 2024 17:21:51 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi027.front.sepia.ceph.com:8000/sbnvju-64/MULTIPARTsbnvju-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709572909.772492_857.1","last_update":"2024-03-04T17:21:49.772492Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:51.839+0000 7fdd48a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:51.840+0000 7fdd48a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:51Z"} 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:51Z"} 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:56.968+0000 7f5940911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:21:56.968+0000 7f5940911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:56Z"} 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:21:56Z"} 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:02.099+0000 7f6b42412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:02.100+0000 7f6b42412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:02Z"} 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:02Z"} 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:07.224+0000 7ff4f6a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:07.225+0000 7ff4f6a67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:07Z"} 2024-03-04T19:09:56.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:07Z"} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:12.350+0000 7f2e7e530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:12.351+0000 7f2e7e530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:12Z"} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.840554_431.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.840554Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:12Z"} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.840554_431.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.840554Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.840554_431.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.840554_431.1 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:17.582+0000 7ff8fb933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:17.583+0000 7ff8fb933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:17Z"} 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:17Z"} 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:22.729+0000 7fbc03630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:22.730+0000 7fbc03630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:22Z"} 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.843902_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.843902Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:22Z"} 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.843902_433.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.843902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.843902_433.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.843902_433.1 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:27.876+0000 7f4995644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:27.877+0000 7f4995644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:27Z"} 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:27Z"} 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:33.004+0000 7f632b03a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:33.005+0000 7f632b03a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:33Z"} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:33Z"} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:38.133+0000 7fbfc9fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:38.134+0000 7fbfc9fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:38Z"} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.873456_437.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.873456Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:38Z"} 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.873456_437.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.873456Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.873456_437.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.873456_437.1 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:43.292+0000 7f1cc9d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:43.292+0000 7f1cc9d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:43Z"} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:43Z"} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:48.430+0000 7fb32900f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:48.431+0000 7fb32900f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:48Z"} 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.858235_434.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.858235Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:48Z"} 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.858235_434.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.858235Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.858235_434.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.858235_434.1 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:53.563+0000 7f593bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:53.564+0000 7f593bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:53Z"} 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:53Z"} 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:58.699+0000 7f101734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:22:58.699+0000 7f101734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:58Z"} 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.951904_442.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.951904Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:22:58Z"} 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.951904_442.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.951904Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.951904_442.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.951904_442.1 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:03.841+0000 7f1d77812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:03.842+0000 7f1d77812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:03Z"} 2024-03-04T19:09:56.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.934048_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.934048Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:03Z"} 2024-03-04T19:09:56.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.934048_439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.934048Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.934048_439.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.934048_439.1 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:08.970+0000 7f0326c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:08.971+0000 7f0326c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:08Z"} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:08Z"} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:14.095+0000 7fdc43331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:14.096+0000 7fdc43331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:14Z"} 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:14Z"} 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:19.239+0000 7fae5fc2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:19.240+0000 7fae5fc2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:19Z"} 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.860903_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.860903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:19Z"} 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.860903_435.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.860903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.860903_435.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.860903_435.1 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:23:23.517+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:24.364+0000 7f1c38b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:24.365+0000 7f1c38b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:24Z"} 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.841505_432.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.841505Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:24Z"} 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.841505_432.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.841505Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.841505_432.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.841505_432.1 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:29.502+0000 7f5ad1d09800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:29.503+0000 7f5ad1d09800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:29Z"} 2024-03-04T19:09:56.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.969696_444.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.969696Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:29Z"} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.969696_444.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.969696Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.969696_444.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.969696_444.1 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:34.636+0000 7f5f43844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:34.637+0000 7f5f43844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:34Z"} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:34Z"} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:39.820+0000 7f01bf024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:39.833+0000 7f01bf024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:39Z"} 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:39Z"} 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:45.042+0000 7f68b532a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:45.043+0000 7f68b532a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:45Z"} 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:45Z"} 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:50.192+0000 7f61b1931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:50.193+0000 7f61b1931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.047580_518.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.047580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:50Z"} 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.047580_518.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.047580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:50Z"} 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.047580_518.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.047580Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.047580_518.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570680.047580_518.1 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:55.338+0000 7f261ee38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:23:55.339+0000 7f261ee38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:55Z"} 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:23:55Z"} 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:00.464+0000 7fcf16a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:00.465+0000 7fcf16a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:00Z"} 2024-03-04T19:09:56.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:00Z"} 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:05.591+0000 7fb4e4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:05.592+0000 7fb4e4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:05Z"} 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:05Z"} 2024-03-04T19:09:56.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:10.720+0000 7f4d87c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:10.721+0000 7f4d87c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:10Z"} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:10Z"} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:15.849+0000 7fe09d88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:15.850+0000 7fe09d88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:15Z"} 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:15Z"} 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.192640_463.1 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:20.982+0000 7f2df552f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:20.983+0000 7f2df552f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:20Z"} 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:20Z"} 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:24:21.398+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:26.116+0000 7fd5da130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:26.117+0000 7fd5da130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:26Z"} 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:26Z"} 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.193667_464.1 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:31.260+0000 7f2973b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:31.261+0000 7f2973b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:31Z"} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:31Z"} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:36.400+0000 7fd06d72a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:36.401+0000 7fd06d72a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:36Z"} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:36Z"} 2024-03-04T19:09:56.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.160094_460.1 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:41.529+0000 7fdd8fb29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:41.530+0000 7fdd8fb29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:41Z"} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:41Z"} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.155110_458.1 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:46.656+0000 7f871d616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:46.656+0000 7f871d616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:46Z"} 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:46Z"} 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:51.799+0000 7fa011213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:51.799+0000 7fa011213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:51Z"} 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:51Z"} 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:56.937+0000 7fe5fee9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:24:56.938+0000 7fe5fee9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:56Z"} 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:24:56Z"} 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:02.085+0000 7f32ec72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:02.086+0000 7f32ec72f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:02Z"} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:02Z"} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:07.239+0000 7fe9e272b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:07.239+0000 7fe9e272b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:07Z"} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:07Z"} 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:12.388+0000 7f0457a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:12.389+0000 7f0457a9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:12Z"} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:12Z"} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:17.553+0000 7fd663142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:17.554+0000 7fd663142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:17Z"} 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:17Z"} 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:22.781+0000 7f3516e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:22.782+0000 7f3516e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:22Z"} 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:22Z"} 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:27.942+0000 7f3f2612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:27.943+0000 7f3f2612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:27Z"} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:27Z"} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:33.146+0000 7f2130d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:33.147+0000 7f2130d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:33Z"} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:33Z"} 2024-03-04T19:09:56.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:38.393+0000 7f7065c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:38.398+0000 7f7065c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:38Z"} 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:38Z"} 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:43.553+0000 7f7348467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:43.554+0000 7f7348467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:43Z"} 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:43Z"} 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:48.697+0000 7fcd941a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:48.698+0000 7fcd941a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:48Z"} 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:48Z"} 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:53.827+0000 7fd17cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:53.828+0000 7fd17cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:53Z"} 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:53Z"} 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:58.963+0000 7f8573b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:25:58.964+0000 7f8573b28800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:58Z"} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:25:58Z"} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.020506_513.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.020506Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.020506_513.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570680.020506_513.1 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:04.097+0000 7f6b4a411800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:04.098+0000 7f6b4a411800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:04Z"} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:04Z"} 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:09.240+0000 7f262cc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:09.241+0000 7f262cc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:09Z"} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:09Z"} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:14.376+0000 7fd669142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:14.377+0000 7fd669142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:14Z"} 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:14Z"} 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:19.537+0000 7ff1f5c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:19.538+0000 7ff1f5c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:19Z"} 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:19Z"} 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:24.674+0000 7fa96dc13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:24.675+0000 7fa96dc13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:24Z"} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:24Z"} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:29.837+0000 7f2863e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:29.837+0000 7f2863e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:29Z"} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:29Z"} 2024-03-04T19:09:56.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:34.968+0000 7f749492e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:34.969+0000 7f749492e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:34Z"} 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:34Z"} 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:40.114+0000 7f0857d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:40.115+0000 7f0857d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:40Z"} 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:40Z"} 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:45.243+0000 7fc479e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:45.244+0000 7fc479e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:45Z"} 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:45Z"} 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:50.389+0000 7efd24b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:50.390+0000 7efd24b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:50Z"} 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:50Z"} 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:55.536+0000 7f9a7b02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:26:55.537+0000 7f9a7b02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:55Z"} 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:26:55Z"} 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709572909.772492_857.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709572909.772492_857.1 target=1_1709570372.215333_466.1 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2024-03-04T19:09:56.672 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1316, in test_encrypted_object_sync 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-67 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-67/ 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-67/ 2024-03-04T19:09:56.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-67/ 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:27:00 GMT 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:/sbnvju-67/ 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.674 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:v4xeQS2ZcuhEdBwHABazzpCLr+I= 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:v4xeQS2ZcuhEdBwHABazzpCLr+I=', 'Content-Length': '0'} 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003b41eeddde4b6640-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-67/testobj-sse-c 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-67/testobj-sse-c 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-67/testobj-sse-c 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:09:56.675 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:27:00 GMT 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:/sbnvju-67/testobj-sse-c 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:QKS4IhAwJ/BtpozTWRLklJkVuZE= 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:QKS4IhAwJ/BtpozTWRLklJkVuZE='} 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000060c1f752b9a260d6-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-67/testobj-sse-kms 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-67/testobj-sse-kms 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-67/testobj-sse-kms 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-03-04T19:09:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:27:00 GMT 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:/sbnvju-67/testobj-sse-kms 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:PZd62gFyi6c+vVteVmgYLlCe3pI= 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:27:00 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PZd62gFyi6c+vVteVmgYLlCe3pI='} 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6c2Judmp1LTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000e0d8f2cf02f9d59b-0065e60464-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:27:00 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709573220.591929_891.1","last_update":"2024-03-04T17:27:00.591929Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:00.920+0000 7f155de2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:00.921+0000 7f155de2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:00Z"} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:00Z"} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:06.063+0000 7fee36333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:06.064+0000 7fee36333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:06Z"} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:06Z"} 2024-03-04T19:09:56.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:11.213+0000 7f2a0452f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:11.214+0000 7f2a0452f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:11Z"} 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:11Z"} 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:16.345+0000 7f022702f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:16.346+0000 7f022702f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:16Z"} 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:16Z"} 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:21.484+0000 7fe45522f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:21.485+0000 7fe45522f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:21Z"} 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.193667_464.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.193667Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:21Z"} 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.193667_464.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.193667Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.193667_464.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.193667_464.1 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:26.621+0000 7f3610413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:26.622+0000 7f3610413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:26Z"} 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.078213_520.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.078213Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:26Z"} 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.078213_520.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.078213Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.078213_520.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.078213_520.1 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:31.754+0000 7f917c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:31.755+0000 7f917c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:31Z"} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:31Z"} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:36.876+0000 7fd3f994b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:36.877+0000 7fd3f994b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:36Z"} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:36Z"} 2024-03-04T19:09:56.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:42.023+0000 7fa8fea12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:42.024+0000 7fa8fea12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:42Z"} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:42Z"} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:47.151+0000 7fa10c830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:47.152+0000 7fa10c830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:47Z"} 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:47Z"} 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:52.300+0000 7f7617931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:52.301+0000 7f7617931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:52Z"} 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:52Z"} 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:57.441+0000 7ff57c814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:27:57.442+0000 7ff57c814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:57Z"} 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:27:57Z"} 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:02.583+0000 7f87f1632800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:02.584+0000 7f87f1632800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:02Z"} 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:02Z"} 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:07.733+0000 7f07828c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:07.734+0000 7f07828c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:07Z"} 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:07Z"} 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:12.869+0000 7fce18213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:12.870+0000 7fce18213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:12Z"} 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:12Z"} 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:18.008+0000 7ffb8508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:18.009+0000 7ffb8508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:18Z"} 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:18Z"} 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:23.152+0000 7f701ed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:23.152+0000 7f701ed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:23Z"} 2024-03-04T19:09:56.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:23Z"} 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:28.291+0000 7fcbb8e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:28.292+0000 7fcbb8e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:28Z"} 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:28Z"} 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:33.437+0000 7fe375211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:33.438+0000 7fe375211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.021888_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.021888Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:33Z"} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.021888_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.021888Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:33Z"} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.021888_514.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.021888Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.021888_514.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.021888_514.1 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:38.581+0000 7f4a47613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:38.582+0000 7f4a47613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:38Z"} 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:38Z"} 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:43.748+0000 7f0d46524800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:43.749+0000 7f0d46524800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:43Z"} 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:43Z"} 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:48.884+0000 7f9fea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:48.885+0000 7f9fea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:48Z"} 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:48Z"} 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.192640_463.1 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:54.023+0000 7f1ad729d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:54.024+0000 7f1ad729d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:54Z"} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:54Z"} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:59.230+0000 7f2371528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:28:59.231+0000 7f2371528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:59Z"} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:28:59Z"} 2024-03-04T19:09:56.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:04.472+0000 7fed12213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:04.473+0000 7fed12213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:04Z"} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:04Z"} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:09.655+0000 7fb6a5f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:09.656+0000 7fb6a5f27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:09Z"} 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:09Z"} 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:14.781+0000 7f5fd0c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:14.782+0000 7f5fd0c0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:14Z"} 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:14Z"} 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:19.915+0000 7fb7e9630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:19.916+0000 7fb7e9630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:19Z"} 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:19Z"} 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:25.056+0000 7f6065244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:25.057+0000 7f6065244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:25Z"} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:25Z"} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.189913_462.1 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:30.196+0000 7fd337215800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:30.197+0000 7fd337215800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.025016_515.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.025016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:30Z"} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.025016_515.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.025016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:30Z"} 2024-03-04T19:09:56.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.025016_515.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.025016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.025016_515.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.025016_515.1 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:35.331+0000 7fbbd5530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:35.332+0000 7fbbd5530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:35Z"} 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:35Z"} 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:29:39.018+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:40.453+0000 7f65b2d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:40.454+0000 7f65b2d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:40Z"} 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:40Z"} 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.215333_466.1 2024-03-04T19:09:56.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:45.575+0000 7f2eaa6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:45.575+0000 7f2eaa6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:45Z"} 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:45Z"} 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:50.721+0000 7f9010083800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:50.722+0000 7f9010083800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:50Z"} 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:50Z"} 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:55.890+0000 7f45c1f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:29:55.891+0000 7f45c1f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:55Z"} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:29:55Z"} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:01.025+0000 7fb784520800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:01.026+0000 7fb784520800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:01Z"} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:01Z"} 2024-03-04T19:09:56.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:06.165+0000 7f994f212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:06.166+0000 7f994f212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:06Z"} 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:06Z"} 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:11.312+0000 7f227382e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:11.313+0000 7f227382e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:11Z"} 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:11Z"} 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:16.449+0000 7f4306511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:16.450+0000 7f4306511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:16Z"} 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:16Z"} 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:21.574+0000 7f04fba24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:21.575+0000 7f04fba24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:21Z"} 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:21Z"} 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:26.718+0000 7f9e44b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:26.719+0000 7f9e44b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:26Z"} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:26Z"} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:31.865+0000 7fba28f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:31.866+0000 7fba28f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:31Z"} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:31Z"} 2024-03-04T19:09:56.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:37.018+0000 7f7947416800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:37.019+0000 7f7947416800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:37Z"} 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:37Z"} 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:42.176+0000 7f6a16613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:42.182+0000 7f6a16613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:42Z"} 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:42Z"} 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:47.451+0000 7fbcc512e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:47.452+0000 7fbcc512e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:47Z"} 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:47Z"} 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:52.596+0000 7f144ad3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:52.597+0000 7f144ad3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:52Z"} 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:52Z"} 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:57.744+0000 7fd88e28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:30:57.745+0000 7fd88e28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:57Z"} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:30:57Z"} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:02.872+0000 7f93f5711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:02.873+0000 7f93f5711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:02Z"} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:02Z"} 2024-03-04T19:09:56.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:08.018+0000 7f4f75424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:08.019+0000 7f4f75424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:08Z"} 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:08Z"} 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:13.160+0000 7f1523624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:13.160+0000 7f1523624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:13Z"} 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:13Z"} 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:18.304+0000 7fd47ea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:18.306+0000 7fd47ea13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:18Z"} 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:18Z"} 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:23.442+0000 7f0395b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:23.443+0000 7f0395b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:23Z"} 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:23Z"} 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:28.582+0000 7f5feb52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:28.582+0000 7f5feb52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:28Z"} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:28Z"} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:33.712+0000 7f4572939800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:33.713+0000 7f4572939800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:33Z"} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:33Z"} 2024-03-04T19:09:56.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:38.850+0000 7f1d3a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:38.851+0000 7f1d3a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:38Z"} 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.020506_513.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.020506Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:38Z"} 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.020506_513.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.020506Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.020506_513.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570680.020506_513.1 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:43.986+0000 7f76d4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:43.987+0000 7f76d4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:43Z"} 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:43Z"} 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:49.146+0000 7effefb2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:49.147+0000 7effefb2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:49Z"} 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:49Z"} 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:54.293+0000 7f3ce9616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:54.294+0000 7f3ce9616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:54Z"} 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:54Z"} 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:59.436+0000 7f85d6e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:31:59.437+0000 7f85d6e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:59Z"} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:31:59Z"} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:04.591+0000 7f448a311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:04.592+0000 7f448a311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:04Z"} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:04Z"} 2024-03-04T19:09:56.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573220.591929_891.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573220.591929_891.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.729 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1346, in test_bucket_index_log_trim 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1341, in make_test_bucket 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-68 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/ 2024-03-04T19:09:56.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/ 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/ 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:/sbnvju-68/ 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.731 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:KEpGKJ9f2J3R2kBJ6mDUcOcVQwU= 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:KEpGKJ9f2J3R2kBJ6mDUcOcVQwU=', 'Content-Length': '0'} 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d5715aa3b4298ad5-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/ 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/ 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/ 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.732 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:/sbnvju-68/ 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086fd00587c51b916-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/a 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/a 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/a 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:/sbnvju-68/a 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:2CH7G/TWEEbTrwBmPy6bDZ+3Hqg= 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:2CH7G/TWEEbTrwBmPy6bDZ+3Hqg='} 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049c7f6aea709ec37-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/ 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/ 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/ 2024-03-04T19:09:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:/sbnvju-68/ 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e60f5f2261be0d-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/b 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/b 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/b 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:/sbnvju-68/b 2024-03-04T19:09:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Q/x6FuP9XPA0S1uOv2eoQiuHxYQ= 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Q/x6FuP9XPA0S1uOv2eoQiuHxYQ='} 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6db21227d4f0439-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/ 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/ 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/ 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.737 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:/sbnvju-68/ 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046d902ba872188ad-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/c 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/c 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/c 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:/sbnvju-68/c 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:+9Nw/ZyT/FbfCvOuZH7ReimYLjA= 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+9Nw/ZyT/FbfCvOuZH7ReimYLjA='} 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006881186644cef2eb-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/ 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/ 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/ 2024-03-04T19:09:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:/sbnvju-68/ 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA= 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:giajK/c7BpWXpnP5BCiEoQmqNtA=', 'Content-Length': '0'} 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bff49e35a9c7d48-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-68/d 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-68/d 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-68/d 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:32:09 GMT 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:/sbnvju-68/d 2024-03-04T19:09:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:3lXHfAoomqFZ3WzTkeh6aF5RkRA= 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:32:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:3lXHfAoomqFZ3WzTkeh6aF5RkRA='} 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006951e6b1dabb79ba-0065e60599-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:32:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709573529.662099_919.1","last_update":"2024-03-04T17:32:09.662099Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:10.019+0000 7f483f3a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:10.020+0000 7f483f3a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:10Z"} 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:10Z"} 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:15.153+0000 7fd130c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:15.153+0000 7fd130c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:15Z"} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:15Z"} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:20.302+0000 7f02e8c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:20.303+0000 7f02e8c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:20Z"} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:20Z"} 2024-03-04T19:09:56.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:32:21.369+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:25.433+0000 7fedb8812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:25.434+0000 7fedb8812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:25Z"} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:25Z"} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:30.562+0000 7f1aa9030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:30.563+0000 7f1aa9030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:30Z"} 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:30Z"} 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:35.706+0000 7f178310f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:35.707+0000 7f178310f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:35Z"} 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:35Z"} 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:40.829+0000 7f4d30a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:40.830+0000 7f4d30a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:40Z"} 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:40Z"} 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:45.958+0000 7f25451a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:45.959+0000 7f25451a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:45Z"} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:45Z"} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:51.080+0000 7f2459a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:51.081+0000 7f2459a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:51Z"} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:51Z"} 2024-03-04T19:09:56.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:56.215+0000 7efd4453c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:32:56.216+0000 7efd4453c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:56Z"} 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.013079_452.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.013079Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:32:56Z"} 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.013079_452.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.013079Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.013079_452.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.013079_452.1 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:01.361+0000 7f9d5232d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:01.362+0000 7f9d5232d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:01Z"} 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:01Z"} 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:06.492+0000 7ff7aaf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:06.493+0000 7ff7aaf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:06Z"} 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:06Z"} 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:11.644+0000 7fbd22933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:11.645+0000 7fbd22933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:11Z"} 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:11Z"} 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:16.783+0000 7fe79362e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:16.784+0000 7fe79362e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:16Z"} 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:16Z"} 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:21.917+0000 7f9589f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:21.918+0000 7f9589f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:21Z"} 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:21Z"} 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:27.066+0000 7fdbe892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:27.067+0000 7fdbe892a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:27Z"} 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:27Z"} 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.160094_460.1 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:32.213+0000 7fd7eb62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:32.214+0000 7fd7eb62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:32Z"} 2024-03-04T19:09:56.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:32Z"} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:37.344+0000 7f6bd412d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:37.345+0000 7f6bd412d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:37Z"} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:37Z"} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:42.476+0000 7f8c3152f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:42.477+0000 7f8c3152f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:42Z"} 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:42Z"} 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:47.603+0000 7fa1a2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:47.604+0000 7fa1a2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:47Z"} 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:47Z"} 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:52.762+0000 7fc3c2711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:52.763+0000 7fc3c2711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:52Z"} 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:52Z"} 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:57.912+0000 7f643d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:33:57.913+0000 7f643d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:57Z"} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:33:57Z"} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:03.058+0000 7fa22db1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:03.059+0000 7fa22db1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:03Z"} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:03Z"} 2024-03-04T19:09:56.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:08.298+0000 7f580a32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:08.301+0000 7f580a32f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:08Z"} 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:08Z"} 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:13.574+0000 7f249568c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:13.575+0000 7f249568c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:13Z"} 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:13Z"} 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:18.710+0000 7f242313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:18.711+0000 7f242313c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:18Z"} 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.160094_460.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.160094Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:18Z"} 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.160094_460.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.160094Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.160094_460.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.160094_460.1 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:23.848+0000 7f06dc33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:23.849+0000 7f06dc33b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:23Z"} 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:23Z"} 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:28.994+0000 7fc528812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:28.995+0000 7fc528812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:28Z"} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:28Z"} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:34.118+0000 7fa8fab0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:34.119+0000 7fa8fab0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:34Z"} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:34Z"} 2024-03-04T19:09:56.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:39.259+0000 7f0c14430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:39.260+0000 7f0c14430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:39Z"} 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:39Z"} 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:44.405+0000 7ff5f8731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:44.406+0000 7ff5f8731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:44Z"} 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:44Z"} 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:49.535+0000 7fd92368c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:49.536+0000 7fd92368c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:49Z"} 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:49Z"} 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:54.687+0000 7fdd2040f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:54.688+0000 7fdd2040f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:54Z"} 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:54Z"} 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:34:59.065+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:59.842+0000 7f7400024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:34:59.843+0000 7f7400024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:59Z"} 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:34:59Z"} 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:04.985+0000 7fc970b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:04.986+0000 7fc970b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:04Z"} 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:04Z"} 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:10.111+0000 7f29d7b42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:10.112+0000 7f29d7b42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:10Z"} 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:10Z"} 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:15.247+0000 7f872d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:15.248+0000 7f872d639800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:15Z"} 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:15Z"} 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:20.385+0000 7fc8b728a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:20.385+0000 7fc8b728a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:20Z"} 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:20Z"} 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:25.527+0000 7fbb8d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:25.528+0000 7fbb8d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:25Z"} 2024-03-04T19:09:56.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:25Z"} 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:30.669+0000 7fb94fc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:30.670+0000 7fb94fc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:30Z"} 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:30Z"} 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:35.914+0000 7fd25932f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:35.915+0000 7fd25932f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:35Z"} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:35Z"} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:41.146+0000 7f8deef1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:41.147+0000 7f8deef1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:41Z"} 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.155110_458.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.155110Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:41Z"} 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.155110_458.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.155110Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.155110_458.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.155110_458.1 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:46.271+0000 7fd1b5211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:46.272+0000 7fd1b5211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:46Z"} 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:46Z"} 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:51.408+0000 7f9b10224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:51.409+0000 7f9b10224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:51Z"} 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:51Z"} 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:56.535+0000 7f815002e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:35:56.536+0000 7f815002e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:56Z"} 2024-03-04T19:09:56.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.996984_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.996984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:35:56Z"} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.996984_449.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.996984Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.996984_449.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.996984_449.1 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:01.680+0000 7f411e02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:01.681+0000 7f411e02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:01Z"} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:01Z"} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:06.811+0000 7f1faff2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:06.812+0000 7f1faff2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:06Z"} 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:06Z"} 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:11.953+0000 7ffb54b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:11.954+0000 7ffb54b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:11Z"} 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:17.094+0000 7f83dcb33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:17.095+0000 7f83dcb33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.192640_463.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.192640Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:17Z"} 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.192640_463.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.192640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.192640_463.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.192640_463.1 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:22.235+0000 7f6b6c71f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:22.236+0000 7f6b6c71f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.167376_461.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.167376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:22Z"} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.167376_461.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.167376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:22Z"} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.167376_461.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.167376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.167376_461.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.167376_461.1 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:27.388+0000 7f8092c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:27.389+0000 7f8092c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:27Z"} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:27Z"} 2024-03-04T19:09:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:32.558+0000 7f60a5f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:32.560+0000 7f60a5f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:32Z"} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:32Z"} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:37.884+0000 7faa6092e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:37.889+0000 7faa6092e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:37Z"} 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.994542_448.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.994542Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:37Z"} 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.994542_448.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.994542Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.994542_448.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.994542_448.1 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:43.112+0000 7f0752e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:43.113+0000 7f0752e9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:43Z"} 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:43Z"} 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:48.258+0000 7fc43e911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:48.259+0000 7fc43e911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:48Z"} 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.218063_467.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.218063Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:48Z"} 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.218063_467.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.218063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.218063_467.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.218063_467.1 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:53.395+0000 7fbf47933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:53.396+0000 7fbf47933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:53Z"} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:53Z"} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:58.524+0000 7f77f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:36:58.525+0000 7f77f8b0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:58Z"} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.015683_453.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.015683Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:36:58Z"} 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.015683_453.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.015683Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.015683_453.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570370.015683_453.1 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:03.661+0000 7ffa7892d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:03.661+0000 7ffa7892d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:03Z"} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.997952_450.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.997952Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:03Z"} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.997952_450.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.997952Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.997952_450.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.997952_450.1 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:08.794+0000 7f54d0044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:08.795+0000 7f54d0044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:08Z"} 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:08Z"} 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:13.932+0000 7ff51eb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:13.933+0000 7ff51eb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:13Z"} 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.071168_455.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.071168Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:13Z"} 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.071168_455.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.071168Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573529.662099_919.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.071168_455.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573529.662099_919.1 target=1_1709570371.071168_455.1 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-03-04T19:09:56.793 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.794 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.794 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1400, in test_bucket_reshard_index_log_trim 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1395, in make_test_bucket 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=sbnvju-69 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/ 2024-03-04T19:09:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/ 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/ 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.796 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:18 GMT 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:/sbnvju-69/ 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:uTmeewIU9eu7zNZOaiORUicsU2w= 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:uTmeewIU9eu7zNZOaiORUicsU2w=', 'Content-Length': '0'} 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000776ecfb23a5223d9-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/ 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/ 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/ 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:18 GMT 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:/sbnvju-69/ 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:0gOQRVpjKnamawra3k9w8TJ5WL4= 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:0gOQRVpjKnamawra3k9w8TJ5WL4=', 'Content-Length': '0'} 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2ab08452aaf7dfe-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/a 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/a 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/a 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:18 GMT 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:/sbnvju-69/a 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:j00LshalDHcTfKFgRSLR9JkPKzY= 2024-03-04T19:09:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:18 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:j00LshalDHcTfKFgRSLR9JkPKzY='} 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000860e2c2f00b727a2-0065e606ce-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:18 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/ 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/ 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/ 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.800 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:/sbnvju-69/ 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cffe6e7e521a11e7-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/b 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/b 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/b 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:/sbnvju-69/b 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:wz/sYXk1+f4+yFnxV13Rus6cga8= 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:wz/sYXk1+f4+yFnxV13Rus6cga8='} 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086901d2c24984849-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/ 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/ 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/ 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:/sbnvju-69/ 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7e84c518843604-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/c 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/c 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/c 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.804 INFO:tasks.rgw_multisite_tests:/sbnvju-69/c 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:y7nteBr2P47ZX6g5ktSv4YOv8qo= 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:y7nteBr2P47ZX6g5ktSv4YOv8qo='} 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e74c6f357904455a-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/ 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/ 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/ 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.805 INFO:tasks.rgw_multisite_tests:HEAD 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:/sbnvju-69/ 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM= 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:5iXkBQ7nY7hQ2pZPj3LvZy92WYM=', 'Content-Length': '0'} 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa96d85e00556157-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-69/d 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-69/d 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-69/d 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:37:19 GMT 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:/sbnvju-69/d 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:vSzyF2LKNhiCKH+hHR3eCCbENm0= 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 04 Mar 2024 17:37:19 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:vSzyF2LKNhiCKH+hHR3eCCbENm0='} 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd3eaa3545396345-0065e606cf-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Mon, 04 Mar 2024 17:37:19 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709573838.984527_954.1","last_update":"2024-03-04T17:37:18.984527Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:19.401+0000 7f7250d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:19.402+0000 7f7250d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:19Z"} 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:19Z"} 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:24.558+0000 7f1724d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:24.558+0000 7f1724d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:37:24.589+0000 7fc0b13a4640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:24Z"} 2024-03-04T19:09:56.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570371.064903_454.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:31.064903Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:24Z"} 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570371.064903_454.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:31.064903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570371.064903_454.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570371.064903_454.1 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:29.702+0000 7f0002f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:29.703+0000 7f0002f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:29Z"} 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:29Z"} 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:34.834+0000 7f430bf4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:34.834+0000 7f430bf4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:34Z"} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:34Z"} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:39.972+0000 7fb496412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:39.973+0000 7fb496412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:39Z"} 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:39Z"} 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:45.131+0000 7fdbf9b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:45.132+0000 7fdbf9b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:45Z"} 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.088500_457.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.088500Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:45Z"} 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.088500_457.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.088500Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.088500_457.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.088500_457.1 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:50.344+0000 7f9572430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:50.344+0000 7f9572430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:50Z"} 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.084490_456.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.084490Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:50Z"} 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.084490_456.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.084490Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.084490_456.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.084490_456.1 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:55.492+0000 7f415d6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:37:55.493+0000 7f415d6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:55Z"} 2024-03-04T19:09:56.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570370.000296_451.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:30.000296Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:37:55Z"} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570370.000296_451.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:30.000296Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570370.000296_451.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570370.000296_451.1 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:00.621+0000 7fe5d9a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:00.622+0000 7fe5d9a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:00Z"} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:00Z"} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:05.753+0000 7f8cfc611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:05.754+0000 7f8cfc611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:05Z"} 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:05Z"} 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:10.881+0000 7fce33d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:10.882+0000 7fce33d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:10Z"} 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:10Z"} 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:16.007+0000 7fdd28d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:16.008+0000 7fdd28d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:16Z"} 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.189913_462.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.189913Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:16Z"} 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.189913_462.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.189913Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.189913_462.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.189913_462.1 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:21.144+0000 7f1469e67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:21.145+0000 7f1469e67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.221520_469.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.221520Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:21Z"} 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.221520_469.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.221520Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:21Z"} 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.221520_469.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.221520Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.221520_469.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.221520_469.1 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:26.278+0000 7f5c0574b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:26.279+0000 7f5c0574b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:26Z"} 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.972318_445.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.972318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:26Z"} 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.972318_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.972318Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.972318_445.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.972318_445.1 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:31.418+0000 7f76c8844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:31.419+0000 7f76c8844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:31Z"} 2024-03-04T19:09:56.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:31Z"} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.215333_466.1 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:36.559+0000 7f2494f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:36.560+0000 7f2494f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:36Z"} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.215333_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.215333Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:36Z"} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.215333_466.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.215333Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.215333_466.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.215333_466.1 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:41.705+0000 7ff50622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:41.706+0000 7ff50622e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:41Z"} 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.973308_446.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.973308Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:41Z"} 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.973308_446.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.973308Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.973308_446.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.973308_446.1 2024-03-04T19:09:56.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:46.852+0000 7f07d1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:46.853+0000 7f07d1e14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:46Z"} 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570369.975603_447.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:29.975603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:46Z"} 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570369.975603_447.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:29.975603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570369.975603_447.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570369.975603_447.1 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:52.003+0000 7fe2bbc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:52.004+0000 7fe2bbc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:52Z"} 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570372.158768_459.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:39:32.158768Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:52Z"} 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570372.158768_459.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:39:32.158768Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570372.158768_459.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570372.158768_459.1 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:57.144+0000 7fc849333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:38:57.145+0000 7fc849333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:57Z"} 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:38:57Z"} 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:02.286+0000 7f96fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:02.286+0000 7f96fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:02Z"} 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:02Z"} 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:07.413+0000 7fec7fe22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:07.413+0000 7fec7fe22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:07Z"} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:07Z"} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:12.564+0000 7f3ecdf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:12.565+0000 7f3ecdf3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:12Z"} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:12Z"} 2024-03-04T19:09:56.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:17.701+0000 7f3565e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:17.702+0000 7f3565e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:17Z"} 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:17Z"} 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:22.838+0000 7f3af6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:22.839+0000 7f3af6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:22Z"} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:22Z"} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:27.982+0000 7f878cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:27.982+0000 7f878cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:27Z"} 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:27Z"} 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:33.131+0000 7f2b7273c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:33.132+0000 7f2b7273c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:33Z"} 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:33Z"} 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:38.295+0000 7f0033d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:38.297+0000 7f0033d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:38Z"} 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:38Z"} 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.242177_729.1 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:43.595+0000 7f3cea212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:43.596+0000 7f3cea212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:43Z"} 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:43Z"} 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:48.720+0000 7f760eb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:48.721+0000 7f760eb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:48Z"} 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:48Z"} 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:53.849+0000 7fa332016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:53.850+0000 7fa332016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:53Z"} 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:53Z"} 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:58.982+0000 7f638b62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:39:58.983+0000 7f638b62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:58Z"} 2024-03-04T19:09:56.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:39:58Z"} 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi027.stdout: INFO: 2024-03-04T17:40:01.343+0000 7fc0b2ba7640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:04.118+0000 7f5eb5731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:04.119+0000 7f5eb5731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:04Z"} 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:04Z"} 2024-03-04T19:09:56.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:09.254+0000 7f6ed3444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:09.254+0000 7f6ed3444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:09Z"} 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:09Z"} 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:14.383+0000 7fa0d7c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:14.384+0000 7fa0d7c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:14Z"} 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:14Z"} 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:19.527+0000 7f36bb636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:19.528+0000 7f36bb636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:19Z"} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:19Z"} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:24.660+0000 7ff1a47af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:24.661+0000 7ff1a47af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:24Z"} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:24Z"} 2024-03-04T19:09:56.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:29.925+0000 7f636bf24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:29.926+0000 7f636bf24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:29Z"} 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:29Z"} 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:35.124+0000 7fe891122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:35.125+0000 7fe891122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:35Z"} 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:35Z"} 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:40.301+0000 7fe957b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:40.303+0000 7fe957b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:40Z"} 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:40Z"} 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:45.566+0000 7f028dac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:45.567+0000 7f028dac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:45Z"} 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:45Z"} 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:50.786+0000 7f703c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:50.787+0000 7f703c42e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:50Z"} 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:50Z"} 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:55.987+0000 7f240d32b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:40:55.988+0000 7f240d32b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:55Z"} 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:40:55Z"} 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:01.133+0000 7fb55f89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:01.134+0000 7fb55f89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:01Z"} 2024-03-04T19:09:56.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:01Z"} 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:06.277+0000 7f00bc52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:06.277+0000 7f00bc52a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:06Z"} 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:06Z"} 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:11.410+0000 7fc13f933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:11.411+0000 7fc13f933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:11Z"} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:11Z"} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571980.296697_734.1 2024-03-04T19:09:56.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:16.542+0000 7f2e9fc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:16.543+0000 7f2e9fc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:16Z"} 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:16Z"} 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:21.678+0000 7f992043a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:21.678+0000 7f992043a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:21Z"} 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:21Z"} 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:26.821+0000 7ff11342e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:26.822+0000 7ff11342e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:26Z"} 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:26Z"} 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:31.955+0000 7fdba360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:31.956+0000 7fdba360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:31Z"} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:31Z"} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:37.101+0000 7fd6fe20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:37.102+0000 7fd6fe20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:37Z"} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:37Z"} 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:42.236+0000 7ff8d0c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:42.237+0000 7ff8d0c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.754989_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.754989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:42Z"} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.754989_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.754989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:42Z"} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572290.754989_772.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:11:30.754989Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709572290.754989_772.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709572290.754989_772.1 2024-03-04T19:09:56.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:47.364+0000 7fcb18730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:47.365+0000 7fcb18730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:47Z"} 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:47Z"} 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:52.508+0000 7f88a7a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:52.509+0000 7f88a7a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:52Z"} 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:52Z"} 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:57.637+0000 7f829582f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:41:57.638+0000 7f829582f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:57Z"} 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:41:57Z"} 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:02.762+0000 7f6ef6c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:02.763+0000 7f6ef6c8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:02Z"} 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:02Z"} 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:07.909+0000 7f13d081e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:07.910+0000 7f13d081e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:07Z"} 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:07Z"} 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:13.044+0000 7f6a78444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:13.045+0000 7f6a78444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:13Z"} 2024-03-04T19:09:56.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:13Z"} 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:18.200+0000 7f28fd511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:18.202+0000 7f28fd511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:18Z"} 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:18Z"} 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:23.441+0000 7f5e33267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:23.442+0000 7f5e33267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:23Z"} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:23Z"} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709573838.984527_954.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709573838.984527_954.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.862 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.863 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1477, in test_bucket_reshard_incremental 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-70/ 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-70/ 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-70/ 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:42:28 GMT 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:/sbnvju-70/ 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:PfkHByoH3kmkVOqRjY27dxHRKjg= 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:42:28 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:PfkHByoH3kmkVOqRjY27dxHRKjg=', 'Content-Length': '0'} 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005b5285011feb9b34-0065e60804-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:42:28 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-70 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709574148.508071_969.1","last_update":"2024-03-04T17:42:28.508071Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:28.780+0000 7f60eac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:28.781+0000 7f60eac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:28Z"} 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:28Z"} 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:33.906+0000 7f1a0b044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:33.907+0000 7f1a0b044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:33Z"} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:33Z"} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:39.040+0000 7fb2f0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:39.041+0000 7fb2f0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:39Z"} 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:39Z"} 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:44.181+0000 7fdb4e7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:44.182+0000 7fdb4e7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:44Z"} 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:44Z"} 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:49.317+0000 7f4765630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:49.318+0000 7f4765630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:49Z"} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:49Z"} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:54.446+0000 7f899f617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:54.446+0000 7f899f617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:54Z"} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:54Z"} 2024-03-04T19:09:56.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571980.296697_734.1 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:59.588+0000 7f6519d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:42:59.588+0000 7f6519d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:59Z"} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:42:59Z"} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:04.775+0000 7f42b8fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:04.776+0000 7f42b8fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:04Z"} 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:04Z"} 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:09.918+0000 7f5045d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:09.919+0000 7f5045d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:09Z"} 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:09Z"} 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:15.065+0000 7fcd88a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:15.066+0000 7fcd88a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:15Z"} 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:15Z"} 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:20.266+0000 7f19b6342800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:20.267+0000 7f19b6342800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:20Z"} 2024-03-04T19:09:56.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:20Z"} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:25.403+0000 7f485a931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:25.404+0000 7f485a931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:25Z"} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:25Z"} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:30.544+0000 7fe1a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:30.545+0000 7fe1a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:30Z"} 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:30Z"} 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:56.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:35.694+0000 7fcca8d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:35.695+0000 7fcca8d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:35Z"} 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:35Z"} 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:40.832+0000 7f64f9726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:40.833+0000 7f64f9726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:40Z"} 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:40Z"} 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:45.958+0000 7fa1ab411800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:45.959+0000 7fa1ab411800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:45Z"} 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:45Z"} 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:51.085+0000 7fed23726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:51.086+0000 7fed23726800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:51Z"} 2024-03-04T19:09:56.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:51Z"} 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:56.229+0000 7f1b5de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:43:56.230+0000 7f1b5de44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:56Z"} 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:43:56Z"} 2024-03-04T19:09:56.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:01.381+0000 7f0cb5533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:01.382+0000 7f0cb5533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:01Z"} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:01Z"} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:06.527+0000 7fd55b12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:06.528+0000 7fd55b12a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:06Z"} 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:06Z"} 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:11.662+0000 7f7dcf015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:11.663+0000 7f7dcf015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:11Z"} 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:11Z"} 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:16.805+0000 7f9e86e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:16.805+0000 7f9e86e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:16Z"} 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:16Z"} 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:22.024+0000 7ff3f9e85800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:22.029+0000 7ff3f9e85800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:22Z"} 2024-03-04T19:09:56.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:22Z"} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:27.221+0000 7f2993526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:27.227+0000 7f2993526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:27Z"} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:27Z"} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:32.438+0000 7fd3eda24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:32.438+0000 7fd3eda24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:32Z"} 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:32Z"} 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:37.568+0000 7f45d988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:37.569+0000 7f45d988c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:37Z"} 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:37Z"} 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:42.701+0000 7fb0d3011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:42.702+0000 7fb0d3011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:42Z"} 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:42Z"} 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:47.832+0000 7fc5ad92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:47.833+0000 7fc5ad92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:47Z"} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:47Z"} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:52.957+0000 7f5ac1731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:52.958+0000 7f5ac1731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:52Z"} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:52Z"} 2024-03-04T19:09:56.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:58.092+0000 7fba8bc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:44:58.093+0000 7fba8bc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:58Z"} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:44:58Z"} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:03.228+0000 7f9d9e067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:03.229+0000 7f9d9e067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:03Z"} 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:03Z"} 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:08.358+0000 7f3378e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:08.359+0000 7f3378e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:08Z"} 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:08Z"} 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:13.495+0000 7fa2e0c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:13.496+0000 7fa2e0c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:13Z"} 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:13Z"} 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:18.637+0000 7fbad8630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:18.639+0000 7fbad8630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:18Z"} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:18Z"} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:23.775+0000 7fa72ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:23.776+0000 7fa72ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:23Z"} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:23Z"} 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:28.920+0000 7f4286b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:28.921+0000 7f4286b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:28Z"} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:28Z"} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:34.071+0000 7fda4531e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:34.072+0000 7fda4531e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:34Z"} 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:34Z"} 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:39.223+0000 7f7e70214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:39.224+0000 7f7e70214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:39Z"} 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:39Z"} 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:44.421+0000 7f49bd430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:44.422+0000 7f49bd430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:44Z"} 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:49.724+0000 7febaaa10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:49.725+0000 7febaaa10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:49Z"} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:54.867+0000 7f0688644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:45:54.868+0000 7f0688644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:45:54Z"} 2024-03-04T19:09:56.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:00.003+0000 7f8417e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:00.004+0000 7f8417e12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:00Z"} 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:00Z"} 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:05.142+0000 7fddd7c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:05.142+0000 7fddd7c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:05Z"} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:05Z"} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:10.275+0000 7fde5882e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:10.276+0000 7fde5882e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:10Z"} 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:10Z"} 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:15.398+0000 7fdeba69d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:15.399+0000 7fdeba69d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:15Z"} 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:15Z"} 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:20.564+0000 7f7bbed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:20.565+0000 7f7bbed33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:20Z"} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:20Z"} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:25.726+0000 7f3e89ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:25.726+0000 7f3e89ac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:25Z"} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:25Z"} 2024-03-04T19:09:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:30.879+0000 7fec1ad31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:30.880+0000 7fec1ad31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:30Z"} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:30Z"} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:36.023+0000 7fd6a0c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:36.024+0000 7fd6a0c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:36Z"} 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:36Z"} 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:41.149+0000 7f5c32c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:41.150+0000 7f5c32c11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:41Z"} 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:41Z"} 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:46.292+0000 7ff415c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:46.293+0000 7ff415c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:46Z"} 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:46Z"} 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:51.435+0000 7f6d6db2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:51.435+0000 7f6d6db2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:51Z"} 2024-03-04T19:09:56.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:51Z"} 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:56.562+0000 7f7e44b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:46:56.563+0000 7f7e44b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:56Z"} 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:46:56Z"} 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:01.699+0000 7fea35830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:01.700+0000 7fea35830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:01Z"} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:01Z"} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:06.822+0000 7fe7dea11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:06.823+0000 7fe7dea11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:06Z"} 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:06Z"} 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:11.955+0000 7fb946d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:11.955+0000 7fb946d35800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:11Z"} 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:11Z"} 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:17.104+0000 7f9da5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:17.105+0000 7f9da5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:17Z"} 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:17Z"} 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:22.243+0000 7f2232444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:22.244+0000 7f2232444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:22Z"} 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:22Z"} 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:27.368+0000 7fbd83415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:27.369+0000 7fbd83415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:27Z"} 2024-03-04T19:09:56.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:27Z"} 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:32.493+0000 7fe567842800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:32.494+0000 7fe567842800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:32Z"} 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:32Z"} 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574148.508071_969.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574148.508071_969.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.916 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1507, in test_bucket_reshard_full 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-71/ 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-71/ 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-71/ 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:47:37 GMT 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:/sbnvju-71/ 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:56.918 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:d48wkmXAtCIOizaBXHoP30ObX/w= 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:47:37 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d48wkmXAtCIOizaBXHoP30ObX/w=', 'Content-Length': '0'} 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000062e6309f616d18a5-0065e60939-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:47:37 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-71 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709574457.544617_986.1","last_update":"2024-03-04T17:47:37.544617Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:37.896+0000 7f681d815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:37.896+0000 7f681d815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:37Z"} 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:37Z"} 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:43.121+0000 7f1f7852f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:43.122+0000 7f1f7852f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:43Z"} 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:43Z"} 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:48.279+0000 7fe0e9e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:48.280+0000 7fe0e9e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:48Z"} 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:48Z"} 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:53.416+0000 7f1c7a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:53.417+0000 7f1c7a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:53Z"} 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:53Z"} 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:58.544+0000 7fd25cd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:47:58.545+0000 7fd25cd2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:58Z"} 2024-03-04T19:09:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:47:58Z"} 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:03.675+0000 7fbaf832c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:03.676+0000 7fbaf832c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:03Z"} 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:03Z"} 2024-03-04T19:09:56.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:08.815+0000 7fd88392b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:08.816+0000 7fd88392b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:08Z"} 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:08Z"} 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:13.966+0000 7f7784012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:13.967+0000 7f7784012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:13Z"} 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:13Z"} 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:19.106+0000 7f7a0f54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:19.107+0000 7f7a0f54b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:19Z"} 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:19Z"} 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:24.261+0000 7f8817e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:24.262+0000 7f8817e1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:24Z"} 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:24Z"} 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.175444_719.1 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:29.392+0000 7fe2ebb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:29.393+0000 7fe2ebb2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:29Z"} 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:29Z"} 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:34.525+0000 7f20eef26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:34.526+0000 7f20eef26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:34Z"} 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:34Z"} 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:39.664+0000 7f73bcf33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:39.665+0000 7f73bcf33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:39Z"} 2024-03-04T19:09:56.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:39Z"} 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:44.931+0000 7f5ad3024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:44.935+0000 7f5ad3024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:44Z"} 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:44Z"} 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:50.105+0000 7f0b1fec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:50.106+0000 7f0b1fec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:50Z"} 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:50Z"} 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:55.243+0000 7f6c00d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:48:55.244+0000 7f6c00d30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:55Z"} 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:48:55Z"} 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:00.388+0000 7fda2a214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:00.389+0000 7fda2a214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:00Z"} 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:00Z"} 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:05.514+0000 7f5335337800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:05.515+0000 7f5335337800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:05Z"} 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:05Z"} 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:10.644+0000 7fecdb51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:10.645+0000 7fecdb51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:10Z"} 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:10Z"} 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:15.787+0000 7f3c2a72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:15.788+0000 7f3c2a72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:15Z"} 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:15Z"} 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:20.928+0000 7f124ce24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:20.929+0000 7f124ce24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:20Z"} 2024-03-04T19:09:56.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:20Z"} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:26.063+0000 7f4808d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:26.064+0000 7f4808d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:26Z"} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:26Z"} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:31.184+0000 7f840c52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:31.185+0000 7f840c52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:31Z"} 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:31Z"} 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:36.333+0000 7f7fbb414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:36.334+0000 7f7fbb414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:36Z"} 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:36Z"} 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:41.473+0000 7efcebb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:41.474+0000 7efcebb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:41Z"} 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:41Z"} 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:46.605+0000 7f98dda44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:46.606+0000 7f98dda44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:46Z"} 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:46Z"} 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:51.738+0000 7f1ea7b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:51.740+0000 7f1ea7b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:51Z"} 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:51Z"} 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:56.909+0000 7f420472f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:49:56.910+0000 7f420472f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572600.174134_809.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:16:40.174134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:56Z"} 2024-03-04T19:09:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572600.174134_809.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:16:40.174134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:49:56Z"} 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572600.174134_809.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:16:40.174134Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709572600.174134_809.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709572600.174134_809.1 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:02.042+0000 7fa77dd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:02.043+0000 7fa77dd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:02Z"} 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:02Z"} 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:07.194+0000 7fc37f6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:07.195+0000 7fc37f6c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:07Z"} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:07Z"} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:12.335+0000 7f5d9f522800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:12.336+0000 7f5d9f522800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:12Z"} 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:12Z"} 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:17.476+0000 7f98c814b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:17.477+0000 7f98c814b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:17Z"} 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:17Z"} 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:22.607+0000 7f68fc62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:22.608+0000 7f68fc62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:22Z"} 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:22Z"} 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:27.851+0000 7f787ae10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:27.852+0000 7f787ae10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:27Z"} 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:27Z"} 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:33.017+0000 7f2cb9b20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:33.018+0000 7f2cb9b20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:33Z"} 2024-03-04T19:09:56.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:33Z"} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:38.185+0000 7f2f5ad0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:38.186+0000 7f2f5ad0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:38Z"} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:43.327+0000 7f56768c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:43.328+0000 7f56768c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:43Z"} 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:48.470+0000 7fc841a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:48.471+0000 7fc841a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:48Z"} 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:48Z"} 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:53.624+0000 7f7f2992f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:53.625+0000 7f7f2992f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:53Z"} 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:53Z"} 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:58.760+0000 7fa4b6d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:50:58.761+0000 7fa4b6d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:58Z"} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:50:58Z"} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:03.882+0000 7f0dd9013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:03.883+0000 7f0dd9013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:03Z"} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:03Z"} 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:09.021+0000 7f6737213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:09.022+0000 7f6737213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:09Z"} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:09Z"} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:14.151+0000 7fa77fcc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:14.151+0000 7fa77fcc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:14Z"} 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:14Z"} 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:56.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:19.288+0000 7ff874a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:19.288+0000 7ff874a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:19Z"} 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:19Z"} 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:24.411+0000 7fe55f122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:24.412+0000 7fe55f122800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:24Z"} 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:24Z"} 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:29.534+0000 7f7f709a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:29.535+0000 7f7f709a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:29Z"} 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:29Z"} 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:34.660+0000 7f61107a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:34.661+0000 7f61107a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:34Z"} 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:34Z"} 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:39.782+0000 7f801d02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:39.783+0000 7f801d02d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:39Z"} 2024-03-04T19:09:56.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:39Z"} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:44.927+0000 7f71b9230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:44.928+0000 7f71b9230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:44Z"} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:44Z"} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:50.071+0000 7f2219b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:50.072+0000 7f2219b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:50Z"} 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:50Z"} 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:55.199+0000 7fe33d08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:51:55.200+0000 7fe33d08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:55Z"} 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:51:55Z"} 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:00.330+0000 7ff2242c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:00.331+0000 7ff2242c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:00Z"} 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:00Z"} 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:05.480+0000 7fd299d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:05.481+0000 7fd299d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:05Z"} 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:05Z"} 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:10.615+0000 7f4d79c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:10.616+0000 7f4d79c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:10Z"} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:10Z"} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:15.767+0000 7fab3c71e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:15.768+0000 7fab3c71e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:15Z"} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:15Z"} 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:20.893+0000 7f73c542e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:20.894+0000 7f73c542e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:20Z"} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:20Z"} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:26.034+0000 7f24b0013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:26.035+0000 7f24b0013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:26Z"} 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:26Z"} 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:31.177+0000 7f38e1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:31.178+0000 7f38e1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:31Z"} 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:31Z"} 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:36.321+0000 7f9c5ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:36.322+0000 7f9c5ee2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:36Z"} 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:36Z"} 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:41.459+0000 7f45ebc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:41.460+0000 7f45ebc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:41Z"} 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:41Z"} 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2024-03-04T19:09:56.971 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1540, in test_bucket_creation_time 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709574457.544617_986.1","last_update":"2024-03-04T17:47:37.544617Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:46.738+0000 7f20b2f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:46.739+0000 7f20b2f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:46Z"} 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:46Z"} 2024-03-04T19:09:56.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:51.884+0000 7f4337e22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:51.884+0000 7f4337e22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:51Z"} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:51Z"} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:57.026+0000 7f5c1493c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:52:57.027+0000 7f5c1493c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:57Z"} 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:52:57Z"} 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:02.159+0000 7f288f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:02.160+0000 7f288f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:02Z"} 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:02Z"} 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:07.285+0000 7fb53243a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:07.286+0000 7fb53243a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:07Z"} 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:07Z"} 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:12.415+0000 7fae27339800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:12.415+0000 7fae27339800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:12Z"} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:12Z"} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:17.549+0000 7fac0072a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:17.549+0000 7fac0072a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:17Z"} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:17Z"} 2024-03-04T19:09:56.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:22.692+0000 7fdb1ea9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:22.693+0000 7fdb1ea9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:22Z"} 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:22Z"} 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:27.845+0000 7fd2e2c34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:27.846+0000 7fd2e2c34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:27Z"} 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:27Z"} 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:32.979+0000 7f392bb22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:32.980+0000 7f392bb22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:32Z"} 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:32Z"} 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:38.115+0000 7f282252f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:38.116+0000 7f282252f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:38Z"} 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:38Z"} 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:43.236+0000 7ff1bca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:43.237+0000 7ff1bca30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:43Z"} 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:43Z"} 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:48.387+0000 7ffb67130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:48.388+0000 7ffb67130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:48Z"} 2024-03-04T19:09:56.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:48Z"} 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:53.537+0000 7f6a59813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:53.538+0000 7f6a59813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:53Z"} 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:53Z"} 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:58.685+0000 7ffaa6830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:53:58.686+0000 7ffaa6830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:53:58Z"} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:56.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:03.826+0000 7f415b330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:03.826+0000 7f415b330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:03Z"} 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:03Z"} 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:09.053+0000 7f8720e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:09.054+0000 7f8720e0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:09Z"} 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:09Z"} 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:14.277+0000 7f9983a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:14.278+0000 7f9983a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:14Z"} 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:14Z"} 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:19.427+0000 7fceddb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:19.428+0000 7fceddb4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:19Z"} 2024-03-04T19:09:56.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:19Z"} 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:24.564+0000 7f0f8761e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:24.564+0000 7f0f8761e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:24Z"} 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:24Z"} 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:29.698+0000 7fbedec10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:29.699+0000 7fbedec10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:29Z"} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:29Z"} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:34.831+0000 7f9b86509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:34.832+0000 7f9b86509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:34Z"} 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:34Z"} 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:39.959+0000 7f2b02b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:39.960+0000 7f2b02b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:39Z"} 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:39Z"} 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:45.095+0000 7feba743c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:45.096+0000 7feba743c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:45Z"} 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:45Z"} 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:50.229+0000 7f704fac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:50.230+0000 7f704fac0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:50Z"} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:50Z"} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:55.357+0000 7fc966830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:54:55.358+0000 7fc966830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:55Z"} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:54:55Z"} 2024-03-04T19:09:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:00.480+0000 7f705208c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:00.481+0000 7f705208c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:00Z"} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:00Z"} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:05.622+0000 7f8c2e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:05.623+0000 7f8c2e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:05Z"} 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:05Z"} 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:10.763+0000 7efdf242e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:10.764+0000 7efdf242e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:10Z"} 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:10Z"} 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:15.891+0000 7f38a652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:15.892+0000 7f38a652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:15Z"} 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:15Z"} 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.125073_710.1 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:21.018+0000 7f965af2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:21.019+0000 7f965af2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:21Z"} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:21Z"} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:26.160+0000 7f1c74728800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:26.161+0000 7f1c74728800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:26Z"} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:26Z"} 2024-03-04T19:09:56.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:31.302+0000 7fcd42309800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:31.303+0000 7fcd42309800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:31Z"} 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:31Z"} 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:36.463+0000 7f3d30542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:36.464+0000 7f3d30542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:36Z"} 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:36Z"} 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:41.634+0000 7f488a31f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:41.637+0000 7f488a31f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:41Z"} 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:41Z"} 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:46.910+0000 7f6121f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:46.911+0000 7f6121f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:46Z"} 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:46Z"} 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:52.136+0000 7fdc3f52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:52.137+0000 7fdc3f52d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:52Z"} 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:52Z"} 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:57.270+0000 7fc9ff531800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:55:57.271+0000 7fc9ff531800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:57Z"} 2024-03-04T19:09:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:55:57Z"} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:02.408+0000 7f215ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:02.408+0000 7f215ada2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:02Z"} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:02Z"} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:07.538+0000 7febbc410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:07.539+0000 7febbc410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:07Z"} 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:07Z"} 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:12.681+0000 7f40e3a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:12.682+0000 7f40e3a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:12Z"} 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:12Z"} 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:17.817+0000 7f5f0ab31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:17.818+0000 7f5f0ab31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:17Z"} 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:17Z"} 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:22.962+0000 7fed75f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:22.962+0000 7fed75f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:22Z"} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:22Z"} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:28.087+0000 7f57f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:28.088+0000 7f57f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:28Z"} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:28Z"} 2024-03-04T19:09:57.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:33.224+0000 7f2d82c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:33.225+0000 7f2d82c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:33Z"} 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:33Z"} 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:38.356+0000 7faf39537800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:38.357+0000 7faf39537800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:38Z"} 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:38Z"} 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:43.561+0000 7fc1eb038800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:43.571+0000 7fc1eb038800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:43Z"} 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:43Z"} 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:48.748+0000 7fe471424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:48.749+0000 7fe471424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:48Z"} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:48Z"} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:53.874+0000 7f6e3d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:53.875+0000 7f6e3d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:53Z"} 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:53Z"} 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:59.001+0000 7f171ab2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:56:59.002+0000 7f171ab2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:59Z"} 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:56:59Z"} 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:04.142+0000 7f1deba1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:04.143+0000 7f1deba1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:04Z"} 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:04Z"} 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:09.271+0000 7f5df1a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:09.271+0000 7f5df1a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:09Z"} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:09Z"} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:14.395+0000 7f9220a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:14.396+0000 7f9220a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:14Z"} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:14Z"} 2024-03-04T19:09:57.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:19.515+0000 7fa85393c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:19.516+0000 7fa85393c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:19Z"} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:19Z"} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:24.648+0000 7f99993af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:24.649+0000 7f99993af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:24Z"} 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:24Z"} 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:29.789+0000 7f8acd330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:29.790+0000 7f8acd330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:29Z"} 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:29Z"} 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:34.939+0000 7fc69f611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:34.940+0000 7fc69f611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:34Z"} 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:34Z"} 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:40.077+0000 7f57d3930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:40.078+0000 7f57d3930800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:40Z"} 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:40Z"} 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709571978.154969_713.1 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:45.216+0000 7f0427024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:45.217+0000 7f0427024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:45Z"} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:45Z"} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:50.374+0000 7f0669611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:50.375+0000 7f0669611800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:50Z"} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:50Z"} 2024-03-04T19:09:57.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709574457.544617_986.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709574457.544617_986.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2024-03-04T19:09:57.024 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1625, in test_bucket_sync_run_basic_incremental 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-72/ 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-72/ 2024-03-04T19:09:57.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-72/ 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 17:57:55 GMT 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:/sbnvju-72/ 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.026 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:XhcLW6MqqyHr8lloCt8aEigoMLk= 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 17:57:55 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:XhcLW6MqqyHr8lloCt8aEigoMLk=', 'Content-Length': '0'} 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005c1839f504aa84e1-0065e60ba3-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 17:57:55 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-72 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709575075.434129_1008.1","last_update":"2024-03-04T17:57:55.434129Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:55.760+0000 7f66f3f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:57:55.761+0000 7f66f3f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:55Z"} 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:57:55Z"} 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:00.915+0000 7f139f12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:00.916+0000 7f139f12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:00Z"} 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:00Z"} 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:06.062+0000 7fcea401b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:06.062+0000 7fcea401b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:06Z"} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:06Z"} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:11.205+0000 7f836992d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:11.206+0000 7f836992d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:11Z"} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:11Z"} 2024-03-04T19:09:57.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:16.327+0000 7f0677c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:16.327+0000 7f0677c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:16Z"} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:16Z"} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:21.470+0000 7f12d5c10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:21.470+0000 7f12d5c10800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:21Z"} 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:21Z"} 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:26.601+0000 7f08e6524800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:26.601+0000 7f08e6524800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:26Z"} 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:26Z"} 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:31.734+0000 7f63f3642800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:31.735+0000 7f63f3642800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:31Z"} 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:31Z"} 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:36.874+0000 7f82c6c38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:36.875+0000 7f82c6c38800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:36Z"} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:36Z"} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:42.017+0000 7fcbb8c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:42.018+0000 7fcbb8c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:42Z"} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:42Z"} 2024-03-04T19:09:57.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:47.152+0000 7fb534e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:47.153+0000 7fb534e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:47Z"} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:47Z"} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:52.407+0000 7f92777a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:52.408+0000 7f92777a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:52Z"} 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:52Z"} 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:57.565+0000 7f8373709800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:58:57.566+0000 7f8373709800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:57Z"} 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:58:57Z"} 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:02.705+0000 7fdb4e92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:02.706+0000 7fdb4e92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:02Z"} 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:02Z"} 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:07.841+0000 7f711388c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:07.842+0000 7f711388c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:07Z"} 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:07Z"} 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:12.965+0000 7fab5c813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:12.966+0000 7fab5c813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:12Z"} 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:12Z"} 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.121915_709.1 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:18.102+0000 7fb035617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:18.103+0000 7fb035617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:18Z"} 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:18Z"} 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:23.239+0000 7f536162e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:23.239+0000 7f536162e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:23Z"} 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:23Z"} 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:28.390+0000 7f8d9e444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:28.391+0000 7f8d9e444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:28Z"} 2024-03-04T19:09:57.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:28Z"} 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:33.527+0000 7f4e8880f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:33.528+0000 7f4e8880f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:33Z"} 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:33Z"} 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:38.669+0000 7f0d32e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:38.670+0000 7f0d32e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:38Z"} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:38Z"} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:43.815+0000 7f3b31130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:43.816+0000 7f3b31130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:43Z"} 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:43Z"} 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:48.953+0000 7f6c2ec8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:48.953+0000 7f6c2ec8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:48Z"} 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:48Z"} 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:54.114+0000 7f43bcb37800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:54.115+0000 7f43bcb37800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:54Z"} 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:54Z"} 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:59.250+0000 7fcc55539800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T17:59:59.251+0000 7fcc55539800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:59Z"} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T17:59:59Z"} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:04.443+0000 7f802601b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:04.447+0000 7f802601b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:04Z"} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:04Z"} 2024-03-04T19:09:57.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:09.689+0000 7fa9d5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:09.690+0000 7fa9d5a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:09Z"} 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:09Z"} 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:14.822+0000 7fc78e013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:14.823+0000 7fc78e013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:14Z"} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:14Z"} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:19.946+0000 7f0197509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:19.947+0000 7f0197509800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:19Z"} 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:19Z"} 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:25.081+0000 7fa0cab2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:25.082+0000 7fa0cab2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:25Z"} 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:25Z"} 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:30.225+0000 7f9fee2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:30.226+0000 7f9fee2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:30Z"} 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:30Z"} 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:35.348+0000 7f39e1432800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:35.349+0000 7f39e1432800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:35Z"} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:35Z"} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:40.475+0000 7fa5b6839800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:40.476+0000 7fa5b6839800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:40Z"} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:40Z"} 2024-03-04T19:09:57.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:45.622+0000 7f2017c67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:45.623+0000 7f2017c67800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:45Z"} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:45Z"} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:50.760+0000 7f28aaa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:50.761+0000 7f28aaa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:50Z"} 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:50Z"} 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:55.897+0000 7feba9e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:00:55.898+0000 7feba9e44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:55Z"} 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:00:55Z"} 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:01.031+0000 7fbabc612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:01.032+0000 7fbabc612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:01Z"} 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:01Z"} 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:06.155+0000 7f8db6f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:06.156+0000 7f8db6f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:06Z"} 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:06Z"} 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:11.293+0000 7f5398a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:11.294+0000 7f5398a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:11Z"} 2024-03-04T19:09:57.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:11Z"} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:16.447+0000 7f2a85030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:16.448+0000 7f2a85030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:16Z"} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:16Z"} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:21.580+0000 7f8d4a00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:21.581+0000 7f8d4a00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:21Z"} 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:21Z"} 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:26.733+0000 7f4ae0d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:26.734+0000 7f4ae0d3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:26Z"} 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:26Z"} 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:31.931+0000 7f2d6513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:31.932+0000 7f2d6513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:31Z"} 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:31Z"} 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:37.061+0000 7fc0e9131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:37.062+0000 7fc0e9131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:37Z"} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:37Z"} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:42.203+0000 7fb114d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:42.204+0000 7fb114d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:42Z"} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:42Z"} 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:47.350+0000 7fe8b132d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:47.351+0000 7fe8b132d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:47Z"} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:47Z"} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:52.495+0000 7f773ff3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:52.496+0000 7f773ff3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:52Z"} 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:52Z"} 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:57.637+0000 7f902a12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:01:57.638+0000 7f902a12d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:57Z"} 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:01:57Z"} 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:02.766+0000 7f70a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:02.767+0000 7f70a7131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:02Z"} 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:02Z"} 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:07.921+0000 7f76ae931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:07.921+0000 7f76ae931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:07Z"} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:07Z"} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:13.061+0000 7f6198a2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:13.062+0000 7f6198a2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:13Z"} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:13Z"} 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:18.203+0000 7f58c2f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:18.204+0000 7f58c2f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:18Z"} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:18Z"} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:23.360+0000 7f12f7f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:23.361+0000 7f12f7f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:23Z"} 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:23Z"} 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:28.508+0000 7f6d2b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:28.509+0000 7f6d2b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:28Z"} 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:28Z"} 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:33.667+0000 7fc64c142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:33.668+0000 7fc64c142800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:33Z"} 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:33Z"} 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:38.822+0000 7fa0b17a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:38.823+0000 7fa0b17a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:38Z"} 2024-03-04T19:09:57.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:38Z"} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:43.965+0000 7f67cc430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:43.966+0000 7f67cc430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:43Z"} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:43Z"} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:49.213+0000 7fdd45b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:49.213+0000 7fdd45b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:49Z"} 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:49Z"} 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:54.378+0000 7f0408224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:54.379+0000 7f0408224800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:54Z"} 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:54Z"} 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:59.516+0000 7fbb49e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:02:59.516+0000 7fbb49e24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:59Z"} 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:02:59Z"} 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575075.434129_1008.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575075.434129_1008.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.080 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1688, in test_zap_init_bucket_sync_run 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.081 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-73/ 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-73/ 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-73/ 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.082 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:03:04 GMT 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:/sbnvju-73/ 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:+YLaEyMoacgfCSgheKofs4XRd1Y= 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:03:04 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:+YLaEyMoacgfCSgheKofs4XRd1Y=', 'Content-Length': '0'} 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006e3b541d3932e0b1-0065e60cd8-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:03:04 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-73 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-03-04T19:09:57.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:04.896+0000 7f2788a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:04.898+0000 7f2788a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:04Z"} 2024-03-04T19:09:57.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:04Z"} 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:10.062+0000 7fb08d52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:10.063+0000 7fb08d52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:10Z"} 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:10Z"} 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:15.182+0000 7fd54111f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:15.183+0000 7fd54111f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:15Z"} 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:15Z"} 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:20.327+0000 7fed05c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:20.328+0000 7fed05c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:20Z"} 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:20Z"} 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:25.478+0000 7fb16f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:25.479+0000 7fb16f52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:25Z"} 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:25Z"} 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:30.691+0000 7effe0436800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:30.693+0000 7effe0436800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:30Z"} 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:30Z"} 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:35.906+0000 7f0bdee12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:35.907+0000 7f0bdee12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:35Z"} 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:35Z"} 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:41.136+0000 7f1ab1012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:41.137+0000 7f1ab1012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:41Z"} 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:41Z"} 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:46.335+0000 7f00d580f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:46.336+0000 7f00d580f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:46Z"} 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:46Z"} 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:51.590+0000 7f5e7a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:51.591+0000 7f5e7a30b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:51Z"} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:51Z"} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:56.728+0000 7f20b4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:03:56.729+0000 7f20b4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:56Z"} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:03:56Z"} 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:01.854+0000 7f821f32c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:01.854+0000 7f821f32c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:01Z"} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:01Z"} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:06.990+0000 7f78d4444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:06.991+0000 7f78d4444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:06Z"} 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:06Z"} 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:12.142+0000 7f11c6333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:12.143+0000 7f11c6333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:12Z"} 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:12Z"} 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:17.281+0000 7fe345f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:17.282+0000 7fe345f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:17Z"} 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:17Z"} 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:22.405+0000 7fce05085800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:22.406+0000 7fce05085800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:22Z"} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:22Z"} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:27.546+0000 7f7f88f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:27.547+0000 7f7f88f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:27Z"} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:27Z"} 2024-03-04T19:09:57.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:32.682+0000 7fd951d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:32.683+0000 7fd951d2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:32Z"} 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:32Z"} 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:37.821+0000 7fb63791e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:37.822+0000 7fb63791e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:37Z"} 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:37Z"} 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:42.970+0000 7fc4e8a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:42.971+0000 7fc4e8a14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:42Z"} 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:42Z"} 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:48.110+0000 7f0eeb015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:48.111+0000 7f0eeb015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:48Z"} 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:53.254+0000 7f501ac14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:53.254+0000 7f501ac14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:53Z"} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:53Z"} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:58.389+0000 7f7239f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:04:58.390+0000 7f7239f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:58Z"} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:04:58Z"} 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:03.605+0000 7f056508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:03.606+0000 7f056508c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:03Z"} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:03Z"} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:08.747+0000 7f9cddc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:08.748+0000 7f9cddc14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:08Z"} 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:08Z"} 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:13.883+0000 7fdee4faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:13.884+0000 7fdee4faf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:13Z"} 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:13Z"} 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:19.031+0000 7f5a454c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:19.032+0000 7f5a454c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:19Z"} 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:24.166+0000 7ff88833b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:24.167+0000 7ff88833b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T19:09:57.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:24Z"} 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:29.299+0000 7f443dd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:29.300+0000 7f443dd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:29Z"} 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:34.433+0000 7fd7fc02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:34.434+0000 7fd7fc02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:34Z"} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:34Z"} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:39.565+0000 7f98a6542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:39.566+0000 7f98a6542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:39Z"} 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:39Z"} 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:44.696+0000 7fbb54f1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:44.697+0000 7fbb54f1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:44Z"} 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:44Z"} 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.179124_720.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.179124Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.179124_720.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.179124_720.1 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:49.842+0000 7fe474e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:49.843+0000 7fe474e11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:49Z"} 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:49Z"} 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:55.002+0000 7f1473a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:05:55.003+0000 7f1473a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:05:55Z"} 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:00.151+0000 7f284ef4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:00.152+0000 7f284ef4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:00Z"} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:05.291+0000 7f5e34811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:05.291+0000 7f5e34811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:05Z"} 2024-03-04T19:09:57.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:10.432+0000 7fe6e9b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:10.433+0000 7fe6e9b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:10Z"} 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:15.558+0000 7f0dbc111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:15.559+0000 7f0dbc111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:15Z"} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:15Z"} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:20.702+0000 7f6eebb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:20.703+0000 7f6eebb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:20Z"} 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:20Z"} 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:25.844+0000 7f745c211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:25.845+0000 7f745c211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:25Z"} 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:25Z"} 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:30.973+0000 7f01c9b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:30.974+0000 7f01c9b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:30Z"} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:30Z"} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:36.117+0000 7f4e82333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:36.118+0000 7f4e82333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:36Z"} 2024-03-04T19:09:57.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:41.270+0000 7f1e70f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:41.271+0000 7f1e70f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:41Z"} 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:46.418+0000 7fbc9772a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:46.419+0000 7fbc9772a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:46Z"} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:51.550+0000 7fa682d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:51.550+0000 7fa682d31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:51Z"} 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:51Z"} 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:56.702+0000 7ff86ff2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:06:56.702+0000 7ff86ff2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:56Z"} 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:06:56Z"} 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:01.972+0000 7f5a75ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:01.973+0000 7f5a75ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:01Z"} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:01Z"} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:07.162+0000 7f6538213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:07.163+0000 7f6538213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:07Z"} 2024-03-04T19:09:57.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:12.349+0000 7f568a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:12.350+0000 7f568a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:12Z"} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:17.489+0000 7f2293733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:17.490+0000 7f2293733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:17Z"} 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:22.624+0000 7f02da3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:22.625+0000 7f02da3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:22Z"} 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:27.765+0000 7f134888c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:27.766+0000 7f134888c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:27Z"} 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:32.903+0000 7fc61c022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:32.904+0000 7fc61c022800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:32Z"} 2024-03-04T19:09:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:32Z"} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:38.052+0000 7f305c542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:38.053+0000 7f305c542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:38Z"} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:43.283+0000 7f0c70311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:43.283+0000 7f0c70311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:43Z"} 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:48.412+0000 7f5a66813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:48.413+0000 7f5a66813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:48Z"} 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:53.540+0000 7f8794a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:53.541+0000 7f8794a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:53Z"} 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:58.678+0000 7feaaba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:07:58.679+0000 7feaaba44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:07:58Z"} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:03.826+0000 7f00f8016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:03.827+0000 7f00f8016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:03Z"} 2024-03-04T19:09:57.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:08.984+0000 7f92f602d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:08.985+0000 7f92f602d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:08Z"} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '', 3: ''} 2024-03-04T19:09:57.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_role_sync 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.135 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1735, in test_role_sync 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-sbnvju-1 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-sbnvju-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:host:smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T180814Z 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:b61a4c900621bc06f9dcd08e1f0ed3acdcea5791619daf5a84b2bf995cf880f8 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:20240304T180814Z 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:20240304/front/smithi027/aws4_request 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:3f0369b9e92d6f8fb09d4ede5e37f73ed8e12ab327a32cc0577a82d07ef42010 2024-03-04T19:09:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:c79a4f75a824a89e18a49ffd24bdb8f0e860c8cbc6d4ef44b186aa26fbb5e7e0 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=c79a4f75a824a89e18a49ffd24bdb8f0e860c8cbc6d4ef44b186aa26fbb5e7e0', 'Host': 'smithi027.front.sepia.ceph.com:8000'} 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000016dca28602f1e5d5-0065e60e0e-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'5c62f73f-097e-4a38-aa4f-b2e1b8140933roles-sbnvju-1/arn:aws:iam:::role/roles-sbnvju-12024-03-04T18:08:14.27Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000016dca28602f1e5d5-0065e60e0e-5355-test-zone1' 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone2 name=roles-sbnvju-2 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-sbnvju-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:host:smithi071.front.sepia.ceph.com:8000 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T180814Z 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:d04fcd9ee728b24a7ff09d1b4a3182db06cf2195a3fbf9b6297973719f2dae1f 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:20240304T180814Z 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:20240304/front/smithi071/aws4_request 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:7108251c592d35cd8b1a5fbf5f4cbc5a2b6d0dbcdfacb0041b9f7ad92a7d16c7 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:d50ba8d169de68f46f039a71b8b83c60c8537e2c76f65f8ba051c263fd5e39af 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi071/aws4_request,SignedHeaders=host;x-amz-date,Signature=d50ba8d169de68f46f039a71b8b83c60c8537e2c76f65f8ba051c263fd5e39af', 'Host': 'smithi071.front.sepia.ceph.com:8000'} 2024-03-04T19:09:57.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e4c0c3b343e0e784-0065e60e0e-7072-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'00798078-f3f0-491b-8ab9-2bf858a374f4roles-sbnvju-2/arn:aws:iam:::role/roles-sbnvju-22024-03-04T18:08:14.54Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000e4c0c3b343e0e784-0065e60e0e-7072-test-zone2' 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone3 name=roles-sbnvju-3 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-sbnvju-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:09:57.140 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:host:smithi027.front.sepia.ceph.com:8001 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T180814Z 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:f1a083a55c6b13fc99dc20f05c3d7005f286e4ed7df33639cc55c83b010215ce 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:20240304T180814Z 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:20240304/front/smithi027/aws4_request 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:eed8b8971c6ef7df1fd8a835b8a3fe367c1b412bf987cd9f6716baac0b438580 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:58d00efb17f6a35b0a844711c19974823f93ced6080d84a171eb2b20cb6843c7 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T180814Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=58d00efb17f6a35b0a844711c19974823f93ced6080d84a171eb2b20cb6843c7', 'Host': 'smithi027.front.sepia.ceph.com:8001'} 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000be3e325851c0a769-0065e60e0e-5358-test-zone3'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '730'), ('Date', 'Mon, 04 Mar 2024 18:08:14 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'0666cdb2-02c0-480a-9496-df417fd6630aroles-sbnvju-3/arn:aws:iam:::role/roles-sbnvju-32024-03-04T18:08:14.76Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000be3e325851c0a769-0065e60e0e-5358-test-zone3' 2024-03-04T19:09:57.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709568775.513713_110.1","last_update":"2024-03-04T16:12:55.513713Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:14.304+0000 7f0cf2288800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:14.305+0000 7f0cf2288800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:14Z"} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:19.461+0000 7f6d01a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:19.462+0000 7f6d01a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.179124_720.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.179124Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:19Z"} 2024-03-04T19:09:57.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.179124_720.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.179124Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.179124_720.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.179124_720.1 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:24.612+0000 7f5c7594b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:24.613+0000 7f5c7594b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:24Z"} 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:29.767+0000 7fd6c1e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:29.768+0000 7fd6c1e13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:29Z"} 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:29Z"} 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:34.992+0000 7f5875d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:34.993+0000 7f5875d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:34Z"} 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:34Z"} 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:40.188+0000 7f5fe4b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:40.189+0000 7f5fe4b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:40Z"} 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:45.337+0000 7f79c9926800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:45.337+0000 7f79c9926800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:45Z"} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:50.477+0000 7f40e6011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:50.478+0000 7f40e6011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:50Z"} 2024-03-04T19:09:57.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:55.636+0000 7fc3edc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:08:55.637+0000 7fc3edc2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:08:55Z"} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:00.774+0000 7f02a4530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:00.775+0000 7f02a4530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:00Z"} 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:05.924+0000 7fba74c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:05.925+0000 7fba74c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:05Z"} 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:11.058+0000 7f2609a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:11.059+0000 7f2609a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:11Z"} 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:16.201+0000 7f6371daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:16.201+0000 7f6371daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:16Z"} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:21.325+0000 7f7150212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:21.326+0000 7f7150212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:21Z"} 2024-03-04T19:09:57.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:26.472+0000 7f190de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:26.473+0000 7f190de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:26Z"} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:31.626+0000 7f022c70b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:31.627+0000 7f022c70b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:31Z"} 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:36.775+0000 7f0606824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:36.776+0000 7f0606824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:36Z"} 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:42.035+0000 7fc67d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:42.036+0000 7fc67d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:42Z"} 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:42Z"} 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:47.166+0000 7ff78d630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:47.167+0000 7ff78d630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:47Z"} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:52.317+0000 7f8828e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:52.318+0000 7f8828e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:52Z"} 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:57.484+0000 7f8cc2c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:09:57.485+0000 7f8cc2c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:09:57Z"} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.159 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:02.617+0000 7fa6fb816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:02.618+0000 7fa6fb816800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:02Z"} 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:07.737+0000 7efdb1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:07.738+0000 7efdb1013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:07Z"} 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:12.867+0000 7f23d6731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:12.868+0000 7f23d6731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:12Z"} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:17.989+0000 7fcd46f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:17.990+0000 7fcd46f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:17Z"} 2024-03-04T19:09:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:23.115+0000 7fba3c612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:23.116+0000 7fba3c612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:23Z"} 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:23Z"} 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:28.244+0000 7f5289a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:28.245+0000 7f5289a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:28Z"} 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:33.386+0000 7fcd27410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:33.386+0000 7fcd27410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:33Z"} 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:38.527+0000 7fd624d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:38.528+0000 7fd624d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:38Z"} 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:43.692+0000 7fe29f92c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:43.693+0000 7fe29f92c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:43Z"} 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:48.840+0000 7f5001c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:48.841+0000 7f5001c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:48Z"} 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:53.998+0000 7fe0c1b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:53.999+0000 7fe0c1b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:54Z"} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:54Z"} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:59.161+0000 7f48052c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:10:59.162+0000 7f48052c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:59Z"} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:10:59Z"} 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:04.297+0000 7f2d7428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:04.297+0000 7f2d7428c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:04Z"} 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:04Z"} 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:09.462+0000 7f4cd0d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:09.465+0000 7f4cd0d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:09Z"} 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:14.696+0000 7f3647533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:14.697+0000 7f3647533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:14Z"} 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:19.829+0000 7fdaf8a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:19.829+0000 7fdaf8a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:19Z"} 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:24.962+0000 7f602be0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:24.963+0000 7f602be0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:24Z"} 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:30.099+0000 7fe590b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:30.100+0000 7fe590b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:30Z"} 2024-03-04T19:09:57.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:30Z"} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:35.245+0000 7f81c3533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:35.246+0000 7f81c3533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:35Z"} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:35Z"} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:40.394+0000 7fd2c854b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:40.395+0000 7fd2c854b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:40Z"} 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:40Z"} 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:45.517+0000 7f3417a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:45.518+0000 7f3417a2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:45Z"} 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:45Z"} 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:50.659+0000 7f24dcd3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:50.660+0000 7f24dcd3b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:50Z"} 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:55.800+0000 7fb3caf2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:11:55.801+0000 7fb3caf2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.180479_721.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.180479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.180479_721.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.180479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:11:55Z"} 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.180479_721.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.180479Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.180479_721.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.180479_721.1 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:00.946+0000 7f82b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:00.949+0000 7f82b352d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T19:09:57.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:00Z"} 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:06.191+0000 7fb9b702e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:06.192+0000 7fb9b702e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:06Z"} 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:06Z"} 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:11.347+0000 7fa0c352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:11.348+0000 7fa0c352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:11Z"} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:11Z"} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:16.479+0000 7ff61a615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:16.480+0000 7ff61a615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:16Z"} 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:16Z"} 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:21.621+0000 7f4581624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:21.622+0000 7f4581624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:21Z"} 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:21Z"} 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:26.757+0000 7fb9ea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:26.757+0000 7fb9ea424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:26Z"} 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:31.885+0000 7f9b69326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:31.886+0000 7f9b69326800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:31Z"} 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:37.061+0000 7f5baf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:37.062+0000 7f5baf533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:37Z"} 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:37Z"} 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:42.205+0000 7f415240f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:42.206+0000 7f415240f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:42Z"} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:42Z"} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:47.327+0000 7f911842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:47.328+0000 7f911842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:47Z"} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:47Z"} 2024-03-04T19:09:57.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:52.461+0000 7fd914a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:52.462+0000 7fd914a12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:52Z"} 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:52Z"} 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:57.586+0000 7fad52733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:12:57.587+0000 7fad52733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:57Z"} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:12:57Z"} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:02.712+0000 7f2e1648c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:02.713+0000 7f2e1648c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:02Z"} 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:02Z"} 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:07.955+0000 7febdf88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:07.956+0000 7febdf88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:07Z"} 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:13.098+0000 7fbd6a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:13.099+0000 7fbd6a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:13Z"} 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:13Z"} 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:18.238+0000 7fb437615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:18.239+0000 7fb437615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:18Z"} 2024-03-04T19:09:57.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709568775.513713_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T16:12:55.513713Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:18Z"} 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709568775.513713_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T16:12:55.513713Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709568775.513713_110.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_role_delete_sync 2024-03-04T19:09:57.193 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1751, in test_role_delete_sync 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-sbnvju-4 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-sbnvju-4', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:POST 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:/ 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:host:smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:x-amz-date:20240304T181323Z 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:6288b3b630f5b40a8492676736e698fd56289399f9976b986128f5ed073c2ef1 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.195 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:20240304T181323Z 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:20240304/front/smithi027/aws4_request 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:89a7f99f6174137c51f75b45970cf9eddbd72c1033c261d7de8225d10741b841 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:2160f08a166171f21c6c6bed5e28ec21af78c4255f898ccbb70ebe7596b282e6 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240304T181323Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=4LUC3XJ67HIPQ3U1NKIY/20240304/front/smithi027/aws4_request,SignedHeaders=host;x-amz-date,Signature=2160f08a166171f21c6c6bed5e28ec21af78c4255f898ccbb70ebe7596b282e6', 'Host': 'smithi027.front.sepia.ceph.com:8000'} 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000da1f61449bdbdd2c-0065e60f43-5355-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '731'), ('Date', 'Mon, 04 Mar 2024 18:13:23 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'0a1839a4-a8ef-4bdb-bf5e-1055e6056fb3roles-sbnvju-4/arn:aws:iam:::role/roles-sbnvju-42024-03-04T18:13:23.287Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000da1f61449bdbdd2c-0065e60f43-5355-test-zone1' 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709575384.571956_1018.1","last_update":"2024-03-04T18:03:04.571956Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:23.534+0000 7f6ab1a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:23.537+0000 7f6ab1a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:23Z"} 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:23Z"} 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:28.684+0000 7efdd9f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:28.685+0000 7efdd9f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:28Z"} 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:28Z"} 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.242177_729.1 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:33.930+0000 7f6046c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:33.936+0000 7f6046c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:33Z"} 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:33Z"} 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:39.113+0000 7f1975a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:39.114+0000 7f1975a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:39Z"} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:39Z"} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:44.256+0000 7f5e79730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:44.257+0000 7f5e79730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:44Z"} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:44Z"} 2024-03-04T19:09:57.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:49.389+0000 7f7351f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:49.390+0000 7f7351f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:49Z"} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:49Z"} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:54.516+0000 7fc73f624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:54.517+0000 7fc73f624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:54Z"} 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:54Z"} 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:59.649+0000 7f3b47ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:13:59.650+0000 7f3b47ec0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:59Z"} 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:13:59Z"} 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:04.785+0000 7f1951844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:04.786+0000 7f1951844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:04Z"} 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:04Z"} 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:09.919+0000 7f32fcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:09.919+0000 7f32fcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:09Z"} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:09Z"} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:15.051+0000 7f6b6b013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:15.052+0000 7f6b6b013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:15Z"} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:15Z"} 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:20.189+0000 7ff59c68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:20.190+0000 7ff59c68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:20Z"} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:20Z"} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:25.332+0000 7f3328e42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:25.333+0000 7f3328e42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:25Z"} 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:25Z"} 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:30.466+0000 7f127ac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:30.467+0000 7f127ac0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:30Z"} 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:30Z"} 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:35.599+0000 7f3269f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:35.600+0000 7f3269f1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:35Z"} 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:35Z"} 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:40.728+0000 7f11f4b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:40.729+0000 7f11f4b30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:40Z"} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:40Z"} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:45.874+0000 7fa6e2111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:45.875+0000 7fa6e2111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:45Z"} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:45Z"} 2024-03-04T19:09:57.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:51.005+0000 7fd061d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:51.006+0000 7fd061d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:51Z"} 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:51Z"} 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:56.158+0000 7f5df1212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:14:56.159+0000 7f5df1212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:56Z"} 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:14:56Z"} 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:01.366+0000 7fd06f622800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:01.367+0000 7fd06f622800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:01Z"} 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:01Z"} 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:06.502+0000 7f92bfe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:06.503+0000 7f92bfe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:06Z"} 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:06Z"} 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:11.634+0000 7f498d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:11.635+0000 7f498d434800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:11Z"} 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:11Z"} 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:16.765+0000 7f9f4e244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:16.766+0000 7f9f4e244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:16Z"} 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:16Z"} 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:21.894+0000 7f353b12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:21.895+0000 7f353b12e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:21Z"} 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:21Z"} 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:27.025+0000 7fd04e211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:27.026+0000 7fd04e211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.343580_740.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.343580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:27Z"} 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.343580_740.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.343580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:27Z"} 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.343580_740.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.343580Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.343580_740.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571980.343580_740.1 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:32.155+0000 7f08a3f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:32.156+0000 7f08a3f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:32Z"} 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:32Z"} 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:37.290+0000 7fc4ebba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:37.291+0000 7fc4ebba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:37Z"} 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:37Z"} 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:42.435+0000 7fea12013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:42.436+0000 7fea12013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:42Z"} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:42Z"} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:47.582+0000 7ff2a8614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:47.583+0000 7ff2a8614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:47Z"} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:47Z"} 2024-03-04T19:09:57.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:52.721+0000 7f4f954c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:52.722+0000 7f4f954c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:52Z"} 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:52Z"} 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:57.964+0000 7f570192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:15:57.968+0000 7f570192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:57Z"} 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:15:57Z"} 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:03.121+0000 7f6747a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:03.122+0000 7f6747a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:03Z"} 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:03Z"} 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:08.256+0000 7f973ef42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:08.257+0000 7f973ef42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:08Z"} 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:08Z"} 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:13.406+0000 7f01fc922800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:13.407+0000 7f01fc922800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:13Z"} 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:13Z"} 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:18.544+0000 7fac14b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:18.545+0000 7fac14b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:18Z"} 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:18Z"} 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:23.676+0000 7f1ec7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:23.677+0000 7f1ec7424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:23Z"} 2024-03-04T19:09:57.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:23Z"} 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:28.830+0000 7f6f9b53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:28.831+0000 7f6f9b53b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:28Z"} 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:28Z"} 2024-03-04T19:09:57.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:33.961+0000 7fc537f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:33.962+0000 7fc537f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:33Z"} 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:33Z"} 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:39.106+0000 7efe3d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:39.107+0000 7efe3d14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:39Z"} 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:39Z"} 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:44.240+0000 7f0ba8b21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:44.241+0000 7f0ba8b21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:44Z"} 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:44Z"} 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:49.372+0000 7fa834413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:49.373+0000 7fa834413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:49Z"} 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:49Z"} 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:54.498+0000 7ffa38430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:54.499+0000 7ffa38430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:54Z"} 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:54Z"} 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:59.628+0000 7f4ae309d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:16:59.629+0000 7f4ae309d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:59Z"} 2024-03-04T19:09:57.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:16:59Z"} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:04.752+0000 7fda6b311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:04.753+0000 7fda6b311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:04Z"} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:04Z"} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:09.879+0000 7f693a242800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:09.880+0000 7f693a242800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:09Z"} 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:09Z"} 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:15.036+0000 7f935ea1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:15.037+0000 7f935ea1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:15Z"} 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:15Z"} 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:20.176+0000 7fcba8414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:20.177+0000 7fcba8414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:20Z"} 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:20Z"} 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:25.317+0000 7f188ea8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:25.318+0000 7f188ea8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:25Z"} 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:25Z"} 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:30.458+0000 7fed62613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:30.460+0000 7fed62613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:30Z"} 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:30Z"} 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:35.630+0000 7fb82a68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:35.631+0000 7fb82a68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:35Z"} 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:35Z"} 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:40.839+0000 7f2736015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:40.840+0000 7f2736015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:40Z"} 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:40Z"} 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:45.968+0000 7f401500f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:45.969+0000 7f401500f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:45Z"} 2024-03-04T19:09:57.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:45Z"} 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:51.094+0000 7f9cc5c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:51.095+0000 7f9cc5c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:51Z"} 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:51Z"} 2024-03-04T19:09:57.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:56.237+0000 7f4f0cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:17:56.238+0000 7f4f0cd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:56Z"} 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:17:56Z"} 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:01.390+0000 7f5baec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:01.391+0000 7f5baec13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:01Z"} 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:01Z"} 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:06.528+0000 7f71ce733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:06.529+0000 7f71ce733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:06Z"} 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:06Z"} 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:11.662+0000 7fd54c2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:11.663+0000 7fd54c2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:11Z"} 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:11Z"} 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:16.810+0000 7fdd3a22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:16.811+0000 7fdd3a22e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:16Z"} 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:16Z"} 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:21.951+0000 7f131b92d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:21.952+0000 7f131b92d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:21Z"} 2024-03-04T19:09:57.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:21Z"} 2024-03-04T19:09:57.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:27.077+0000 7f0986739800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:27.078+0000 7f0986739800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:27Z"} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:27Z"} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709575384.571956_1018.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709575384.571956_1018.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.252 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.252 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.252 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.252 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-03-04T19:09:57.252 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1777, in test_bucket_full_sync_after_data_sync_init 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.253 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-74/ 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-74/ 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-74/ 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.254 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:18:32 GMT 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:/sbnvju-74/ 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:VzYo2UVrSAkf8Qn2Tq3Ic+Nxyts= 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:18:32 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:VzYo2UVrSAkf8Qn2Tq3Ic+Nxyts=', 'Content-Length': '0'} 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000db047ca21b169f9e-0065e61078-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:18:32 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-74 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709576312.130624_1055.1","last_update":"2024-03-04T18:18:32.130624Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:32.435+0000 7fbe64d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:32.436+0000 7fbe64d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:32Z"} 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:32Z"} 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:37.584+0000 7f1cdc28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:37.585+0000 7f1cdc28c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:37Z"} 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:37Z"} 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:42.751+0000 7f1276335800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:42.752+0000 7f1276335800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:42Z"} 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:42Z"} 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:47.894+0000 7ff8dc830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:47.895+0000 7ff8dc830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:47Z"} 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:47Z"} 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:53.019+0000 7f508e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:53.020+0000 7f508e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:53Z"} 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:53Z"} 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:58.168+0000 7f82171a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:18:58.169+0000 7f82171a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:58Z"} 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:18:58Z"} 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:03.342+0000 7f57bc08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:03.344+0000 7f57bc08c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:03Z"} 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:03Z"} 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:08.533+0000 7f7fc3424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:08.534+0000 7f7fc3424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:08Z"} 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:08Z"} 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:13.677+0000 7f00e2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:13.678+0000 7f00e2f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:13Z"} 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:13Z"} 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:18.809+0000 7f03f9d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:18.809+0000 7f03f9d11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:18Z"} 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:18Z"} 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:23.942+0000 7f797bda2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:23.943+0000 7f797bda2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:23Z"} 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:23Z"} 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:29.072+0000 7fc3da92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:29.073+0000 7fc3da92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:29Z"} 2024-03-04T19:09:57.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:29Z"} 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:34.221+0000 7f5897267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:34.222+0000 7f5897267800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:34Z"} 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:34Z"} 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:39.359+0000 7f3702931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:39.360+0000 7f3702931800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:39Z"} 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:39Z"} 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:44.495+0000 7f568072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:44.496+0000 7f568072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:44Z"} 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:44Z"} 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:49.631+0000 7f2699813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:49.632+0000 7f2699813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:49Z"} 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:49Z"} 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:54.760+0000 7f60cc0c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:54.761+0000 7f60cc0c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:54Z"} 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:19:54Z"} 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:59.997+0000 7f0c5bc0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:19:59.999+0000 7f0c5bc0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:00Z"} 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:00Z"} 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:05.260+0000 7f6fcf722800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:05.261+0000 7f6fcf722800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:05Z"} 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:05Z"} 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:10.398+0000 7f2c2732f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:10.399+0000 7f2c2732f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:10Z"} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:10Z"} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.125073_710.1 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:15.536+0000 7f339b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:15.537+0000 7f339b815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:15Z"} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:15Z"} 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:20.674+0000 7f695af30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:20.675+0000 7f695af30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:20Z"} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:20Z"} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:25.815+0000 7f12fda0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:25.816+0000 7f12fda0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:25Z"} 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:25Z"} 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:30.954+0000 7f95b0813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:30.955+0000 7f95b0813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:30Z"} 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:30Z"} 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:36.094+0000 7f1492330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:36.095+0000 7f1492330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:36Z"} 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:36Z"} 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:41.231+0000 7fa72612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:41.232+0000 7fa72612f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:41Z"} 2024-03-04T19:09:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:41Z"} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:46.374+0000 7f72b7f0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:46.375+0000 7f72b7f0d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:46Z"} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:46Z"} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:51.506+0000 7fcea5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:51.507+0000 7fcea5733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:51Z"} 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:51Z"} 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:56.635+0000 7f08be49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:20:56.636+0000 7f08be49d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:56Z"} 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:20:56Z"} 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:01.761+0000 7f93e47a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:01.762+0000 7f93e47a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:01Z"} 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:01Z"} 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:06.904+0000 7f4ede667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:06.905+0000 7f4ede667800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:06Z"} 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:06Z"} 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:12.136+0000 7faed4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:12.139+0000 7faed4813800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:12Z"} 2024-03-04T19:09:57.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:12Z"} 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:17.300+0000 7f7f84e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:17.300+0000 7f7f84e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:17Z"} 2024-03-04T19:09:57.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:17Z"} 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:22.437+0000 7fcd9bb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:22.438+0000 7fcd9bb27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:22Z"} 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:22Z"} 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:27.561+0000 7f7adf137800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:27.562+0000 7f7adf137800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:27Z"} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:27Z"} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:32.712+0000 7f236b5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:32.713+0000 7f236b5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:32Z"} 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:32Z"} 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571978.125073_710.1 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:37.848+0000 7f98fff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:37.849+0000 7f98fff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:37Z"} 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:37Z"} 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:42.974+0000 7f69bb030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:42.975+0000 7f69bb030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:42Z"} 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:42Z"} 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:48.105+0000 7fa24a12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:48.106+0000 7fa24a12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:48Z"} 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:48Z"} 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:53.229+0000 7f0c77f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:53.230+0000 7f0c77f2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:53Z"} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:53Z"} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:58.354+0000 7fefde29d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:21:58.355+0000 7fefde29d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:58Z"} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:21:58Z"} 2024-03-04T19:09:57.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:03.479+0000 7fd0f4616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:03.479+0000 7fd0f4616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:03Z"} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:03Z"} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:08.633+0000 7fd4b2a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:08.634+0000 7fd4b2a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:08Z"} 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:08Z"} 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:13.786+0000 7f551908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:13.787+0000 7f551908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:13Z"} 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:13Z"} 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:18.924+0000 7fca66811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:18.925+0000 7fca66811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:18Z"} 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:18Z"} 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:24.199+0000 7f5d3232f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:24.200+0000 7f5d3232f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:24Z"} 2024-03-04T19:09:57.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:24Z"} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:29.352+0000 7f730d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:29.353+0000 7f730d933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:29Z"} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:29Z"} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:34.474+0000 7fdc66e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:34.475+0000 7fdc66e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:34Z"} 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:34Z"} 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:39.606+0000 7fb109a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:39.607+0000 7fb109a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:39Z"} 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:39Z"} 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:44.737+0000 7f14bb439800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:44.738+0000 7f14bb439800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:44Z"} 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:44Z"} 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:49.861+0000 7f5b05a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:49.862+0000 7f5b05a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:49Z"} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:49Z"} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:55.007+0000 7fb66d68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:22:55.008+0000 7fb66d68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:55Z"} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:22:55Z"} 2024-03-04T19:09:57.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:00.164+0000 7f1f21230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:00.165+0000 7f1f21230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:00Z"} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:00Z"} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:05.304+0000 7f233923a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:05.304+0000 7f233923a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:05Z"} 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:05Z"} 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:10.430+0000 7f89f4130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:10.431+0000 7f89f4130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:10Z"} 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:10Z"} 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:15.572+0000 7fcfec52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:15.573+0000 7fcfec52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:15Z"} 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:15Z"} 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:20.709+0000 7f49775a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:20.710+0000 7f49775a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:20Z"} 2024-03-04T19:09:57.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:20Z"} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:25.841+0000 7f82ab528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:25.842+0000 7f82ab528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:25Z"} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:25Z"} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:30.980+0000 7f00285af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:30.981+0000 7f00285af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:30Z"} 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:30Z"} 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:36.111+0000 7f0bd9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:36.112+0000 7f0bd9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:36Z"} 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:36Z"} 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576312.130624_1055.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576312.130624_1055.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.307 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.308 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-75/ 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-75/ 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-75/ 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.309 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:23:41 GMT 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:/sbnvju-75/ 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:Ygm6okJ797HsBifsSfUv8Q4GPMg= 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:23:41 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:Ygm6okJ797HsBifsSfUv8Q4GPMg=', 'Content-Length': '0'} 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000aaf5d1e53579147-0065e611ad-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:23:41 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-75 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709576621.164199_1064.1","last_update":"2024-03-04T18:23:41.164199Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:41.443+0000 7f9271a1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:41.444+0000 7f9271a1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:41Z"} 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:41Z"} 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:46.571+0000 7fe92832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:46.572+0000 7fe92832d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:46Z"} 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:46Z"} 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:51.695+0000 7fdd29016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:51.695+0000 7fdd29016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:51Z"} 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:51Z"} 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:56.817+0000 7fe9e3613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:23:56.818+0000 7fe9e3613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:56Z"} 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:23:56Z"} 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:02.048+0000 7f7aa2933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:02.050+0000 7f7aa2933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:02Z"} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:02Z"} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:07.193+0000 7fd568b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:07.194+0000 7fd568b33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:07Z"} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:07Z"} 2024-03-04T19:09:57.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:12.330+0000 7fc203333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:12.331+0000 7fc203333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:12Z"} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:12Z"} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:17.459+0000 7fc5b7b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:17.460+0000 7fc5b7b2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:17Z"} 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:17Z"} 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:22.712+0000 7fe0de213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:22.713+0000 7fe0de213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:22Z"} 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:22Z"} 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:27.853+0000 7f566aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:27.854+0000 7f566aa2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:27Z"} 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:27Z"} 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:32.988+0000 7fa61e942800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:32.988+0000 7fa61e942800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:32Z"} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:32Z"} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:38.119+0000 7f82ed92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:38.120+0000 7f82ed92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:38Z"} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:38Z"} 2024-03-04T19:09:57.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:43.366+0000 7f3506327800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:43.367+0000 7f3506327800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:43Z"} 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:43Z"} 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:48.509+0000 7fc2bc331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:48.509+0000 7fc2bc331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:48Z"} 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:48Z"} 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:53.652+0000 7fb295d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:53.652+0000 7fb295d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:53Z"} 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:53Z"} 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:58.782+0000 7fa2acb39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:24:58.784+0000 7fa2acb39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:58Z"} 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:24:58Z"} 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:03.906+0000 7fe4c5baf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:03.907+0000 7fe4c5baf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:03Z"} 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:03Z"} 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:09.038+0000 7fb8c2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:09.039+0000 7fb8c2012800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:09Z"} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:09Z"} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:14.183+0000 7f2cf062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:14.183+0000 7f2cf062e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:14Z"} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:14Z"} 2024-03-04T19:09:57.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:19.329+0000 7fb4953a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:19.330+0000 7fb4953a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:19Z"} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:19Z"} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:24.462+0000 7fbb1e52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:24.463+0000 7fbb1e52f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:24Z"} 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:24Z"} 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:29.609+0000 7feba6733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:29.610+0000 7feba6733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:29Z"} 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:29Z"} 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:34.741+0000 7f523d613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:34.742+0000 7f523d613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:34Z"} 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:34Z"} 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:39.885+0000 7fd9c2011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:39.885+0000 7fd9c2011800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:39Z"} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:39Z"} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:45.088+0000 7f1492c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:45.089+0000 7f1492c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:45Z"} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:45Z"} 2024-03-04T19:09:57.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:50.218+0000 7f53a8328800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:50.219+0000 7f53a8328800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:50Z"} 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:50Z"} 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:55.354+0000 7f5803030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:25:55.355+0000 7f5803030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:55Z"} 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:25:55Z"} 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:00.493+0000 7f1c246c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:00.494+0000 7f1c246c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:00Z"} 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:00Z"} 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:05.638+0000 7f08e1836800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:05.639+0000 7f08e1836800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:05Z"} 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:05Z"} 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:10.779+0000 7f836ac83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:10.780+0000 7f836ac83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:10Z"} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:10Z"} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:15.918+0000 7f3f7bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:15.919+0000 7f3f7bfaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:15Z"} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:15Z"} 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:21.072+0000 7fd63b811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:21.073+0000 7fd63b811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:21Z"} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:21Z"} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:26.222+0000 7f4ef6616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:26.223+0000 7f4ef6616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:26Z"} 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:26Z"} 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:31.359+0000 7ff395921800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:31.360+0000 7ff395921800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:31Z"} 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:31Z"} 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:36.498+0000 7f8cd7d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:36.499+0000 7f8cd7d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:36Z"} 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:36Z"} 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:41.727+0000 7f1d97a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:41.728+0000 7f1d97a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:41Z"} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:41Z"} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:46.939+0000 7f9939b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:46.940+0000 7f9939b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:46Z"} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:46Z"} 2024-03-04T19:09:57.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:52.089+0000 7f4c11d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:52.090+0000 7f4c11d2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:52Z"} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:52Z"} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:57.242+0000 7f0f29b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:26:57.243+0000 7f0f29b2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:57Z"} 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:26:57Z"} 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:02.376+0000 7f1b6cb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:02.377+0000 7f1b6cb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:02Z"} 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:02Z"} 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:07.500+0000 7fb83c14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:07.501+0000 7fb83c14b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:07Z"} 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:07Z"} 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:12.645+0000 7f58033a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:12.646+0000 7f58033a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:12Z"} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:12Z"} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:17.799+0000 7f57d180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:17.800+0000 7f57d180f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:17Z"} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:17Z"} 2024-03-04T19:09:57.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:22.935+0000 7fd5f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:22.936+0000 7fd5f608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:22Z"} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:22Z"} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:28.084+0000 7f9bcd40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:28.085+0000 7f9bcd40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:28Z"} 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:28Z"} 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:33.224+0000 7ff339911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:33.225+0000 7ff339911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:33Z"} 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:33Z"} 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:38.371+0000 7f517e68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:38.372+0000 7f517e68c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:38Z"} 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:38Z"} 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:43.517+0000 7f1de3933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:43.517+0000 7f1de3933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:43Z"} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:43Z"} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:48.718+0000 7fcf850c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:48.719+0000 7fcf850c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:48Z"} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:48Z"} 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:53.966+0000 7fa10c93c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:53.967+0000 7fa10c93c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:53Z"} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:53Z"} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:59.170+0000 7f35b314b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:27:59.171+0000 7f35b314b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:59Z"} 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:27:59Z"} 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:04.418+0000 7f5cd949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:04.419+0000 7f5cd949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:04Z"} 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:04Z"} 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:09.541+0000 7f810bba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:09.542+0000 7f810bba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:09Z"} 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:09Z"} 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:14.667+0000 7ff30573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:14.668+0000 7ff30573c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:14Z"} 2024-03-04T19:09:57.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:14Z"} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:19.804+0000 7fd6f8530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:19.805+0000 7fd6f8530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:19Z"} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:19Z"} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:24.943+0000 7f2f8733c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:24.944+0000 7f2f8733c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:24Z"} 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:24Z"} 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:30.076+0000 7fb05e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:30.077+0000 7fb05e424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:30Z"} 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:30Z"} 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:35.201+0000 7f4766d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:35.202+0000 7f4766d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:35Z"} 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:35Z"} 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:40.333+0000 7f613ac20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:40.334+0000 7f613ac20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:40Z"} 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:40Z"} 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:45.454+0000 7f1f3ddaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:45.455+0000 7f1f3ddaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:45Z"} 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:45Z"} 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576621.164199_1064.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576621.164199_1064.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-03-04T19:09:57.361 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1848, in test_bucket_incremental_sync_after_data_sync_init 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-76/ 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-76/ 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-76/ 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:28:50 GMT 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:/sbnvju-76/ 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:TS+OuCdPzCwRhJi0qmb2yzldwH8= 2024-03-04T19:09:57.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:28:50 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:TS+OuCdPzCwRhJi0qmb2yzldwH8=', 'Content-Length': '0'} 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000450192d64099413c-0065e612e2-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:28:50 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-76 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709576930.507437_1081.1","last_update":"2024-03-04T18:28:50.507437Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:50.788+0000 7f0ba6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:50.789+0000 7f0ba6a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:50Z"} 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:50Z"} 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:56.041+0000 7ff2e5d27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:28:56.043+0000 7ff2e5d27800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:56Z"} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:28:56Z"} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:01.296+0000 7fa555c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:01.298+0000 7fa555c2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:01Z"} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:01Z"} 2024-03-04T19:09:57.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:06.443+0000 7ff553d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:06.444+0000 7ff553d4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:06Z"} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:06Z"} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.125073_710.1 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:11.574+0000 7f2b0013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:11.575+0000 7f2b0013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:11Z"} 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:11Z"} 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:16.717+0000 7f4fd4222800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:16.718+0000 7f4fd4222800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:16Z"} 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:16Z"} 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:21.865+0000 7f50ab610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:21.866+0000 7f50ab610800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:21Z"} 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:21Z"} 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:27.029+0000 7f98525a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:27.030+0000 7f98525a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:27Z"} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:27Z"} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:32.308+0000 7f4c1c1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:32.308+0000 7f4c1c1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:32Z"} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:32Z"} 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:37.463+0000 7fdfe4330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:37.464+0000 7fdfe4330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:37Z"} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.613615_561.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.613615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:37Z"} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.613615_561.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.613615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.613615_561.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.613615_561.1 2024-03-04T19:09:57.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:42.634+0000 7ff271330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:42.635+0000 7ff271330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:42Z"} 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:42Z"} 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:47.804+0000 7f5bedd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:47.805+0000 7f5bedd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:47Z"} 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:47Z"} 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:52.958+0000 7fd972d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:52.959+0000 7fd972d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:52Z"} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:52Z"} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:58.096+0000 7fc15f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:29:58.097+0000 7fc15f444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:58Z"} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:29:58Z"} 2024-03-04T19:09:57.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:03.218+0000 7f7f98616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:03.219+0000 7f7f98616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:03Z"} 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:03Z"} 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:08.351+0000 7f0e48c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:08.352+0000 7f0e48c24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:08Z"} 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:08Z"} 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:13.473+0000 7fc2e4c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:13.474+0000 7fc2e4c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:13Z"} 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:13Z"} 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:18.602+0000 7fdf0949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:18.603+0000 7fdf0949d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:18Z"} 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:18Z"} 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:23.721+0000 7f16aef2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:23.722+0000 7f16aef2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:23Z"} 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:23Z"} 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.175444_719.1 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:28.843+0000 7f4bdf135800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:28.844+0000 7f4bdf135800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:28Z"} 2024-03-04T19:09:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:28Z"} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:33.980+0000 7f8e2f330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:33.981+0000 7f8e2f330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:33Z"} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:33Z"} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:39.116+0000 7f886d72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:39.117+0000 7f886d72d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:39Z"} 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:39Z"} 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:44.242+0000 7f82df013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:44.243+0000 7f82df013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:44Z"} 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.081713_521.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.081713Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:44Z"} 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.081713_521.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.081713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.081713_521.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.081713_521.1 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:49.369+0000 7fd663d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:49.370+0000 7fd663d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:49Z"} 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:49Z"} 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:54.502+0000 7f3d9e521800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:54.503+0000 7f3d9e521800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:54Z"} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:54Z"} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:59.657+0000 7fe0a794b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:30:59.658+0000 7fe0a794b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:59Z"} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:30:59Z"} 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:04.798+0000 7ff2f2230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:04.799+0000 7ff2f2230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:04Z"} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:04Z"} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:09.933+0000 7f8c51d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:09.934+0000 7f8c51d39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:09Z"} 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:09Z"} 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:15.074+0000 7f82d9f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:15.075+0000 7f82d9f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:15Z"} 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.616944_562.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.616944Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:15Z"} 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.616944_562.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.616944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.616944_562.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.616944_562.1 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:20.228+0000 7facd4e36800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:20.229+0000 7facd4e36800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:20Z"} 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:20Z"} 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:25.364+0000 7fe404c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:25.365+0000 7fe404c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:25Z"} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.637258_563.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.637258Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:25Z"} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.637258_563.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.637258Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.637258_563.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.637258_563.1 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:30.505+0000 7fea1db31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:30.506+0000 7fea1db31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:30Z"} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:30Z"} 2024-03-04T19:09:57.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:35.679+0000 7f7923f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:35.681+0000 7f7923f11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:35Z"} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:35Z"} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:40.894+0000 7f7dbdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:40.896+0000 7f7dbdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:40Z"} 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:40Z"} 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:46.060+0000 7f906231f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:46.061+0000 7f906231f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:46Z"} 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:46Z"} 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:51.208+0000 7f7846b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:51.209+0000 7f7846b11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:51Z"} 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.608330_559.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.608330Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:51Z"} 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.608330_559.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.608330Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.608330_559.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.608330_559.1 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:56.355+0000 7ff3e8133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:31:56.356+0000 7ff3e8133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:56Z"} 2024-03-04T19:09:57.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:31:56Z"} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:01.495+0000 7f7280724800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:01.496+0000 7f7280724800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:01Z"} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:01Z"} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:06.641+0000 7f8471130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:06.642+0000 7f8471130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:06Z"} 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.612016_560.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.612016Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:06Z"} 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.612016_560.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.612016Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.612016_560.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.612016_560.1 2024-03-04T19:09:57.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:11.780+0000 7fd281f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:11.781+0000 7fd281f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:11Z"} 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.082949_522.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.082949Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:11Z"} 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.082949_522.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.082949Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.082949_522.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.082949_522.1 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:16.930+0000 7fba84039800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:16.931+0000 7fba84039800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:16Z"} 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570680.085880_523.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:44:40.085880Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:16Z"} 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570680.085880_523.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:44:40.085880Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570680.085880_523.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570680.085880_523.1 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:22.076+0000 7f1f6048a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:22.077+0000 7f1f6048a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:22Z"} 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:22Z"} 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:27.212+0000 7f1bb4624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:27.213+0000 7f1bb4624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:27Z"} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:27Z"} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:32.340+0000 7f8782331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:32.341+0000 7f8782331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:32Z"} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:32Z"} 2024-03-04T19:09:57.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:37.497+0000 7f833e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:37.498+0000 7f833e636800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:37Z"} 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:37Z"} 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:42.722+0000 7faa4fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:42.723+0000 7faa4fa11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:42Z"} 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:42Z"} 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:47.922+0000 7f9d73210800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:47.923+0000 7f9d73210800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:47Z"} 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:47Z"} 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.154969_713.1 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:53.069+0000 7fc8d612b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:53.070+0000 7fc8d612b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:53Z"} 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:53Z"} 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.153603_712.1 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:58.205+0000 7fe8e1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:32:58.206+0000 7fe8e1c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:58Z"} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:32:58Z"} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:03.340+0000 7efdad013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:03.341+0000 7efdad013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:03Z"} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:03Z"} 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:08.469+0000 7efe4ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:08.469+0000 7efe4ca14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:08Z"} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:08Z"} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:57.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:13.615+0000 7ff1d031e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:13.616+0000 7ff1d031e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:13Z"} 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:13Z"} 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:18.753+0000 7f7dd192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:18.754+0000 7f7dd192a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:18Z"} 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:18Z"} 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:23.880+0000 7fd78bc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:23.881+0000 7fd78bc16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:23Z"} 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:23Z"} 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:29.017+0000 7fa082c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:29.018+0000 7fa082c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:29Z"} 2024-03-04T19:09:57.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:29Z"} 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:34.214+0000 7f9adc62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:34.217+0000 7f9adc62d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:34Z"} 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:34Z"} 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:39.412+0000 7fbe89a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:39.413+0000 7fbe89a8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:39Z"} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:39Z"} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:44.628+0000 7fbf47817800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:44.629+0000 7fbf47817800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:44Z"} 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:44Z"} 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:49.767+0000 7fcafa526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:49.768+0000 7fcafa526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:49Z"} 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:49Z"} 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:54.913+0000 7f1a00d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:33:54.914+0000 7f1a00d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:54Z"} 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:33:54Z"} 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709576930.507437_1081.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709576930.507437_1081.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-03-04T19:09:57.414 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1886, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-77/ 2024-03-04T19:09:57.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-77/ 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-77/ 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:33:59 GMT 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:/sbnvju-77/ 2024-03-04T19:09:57.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:GvzBw6WI6KoQuRHRFoKcRvYGrBs= 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:33:59 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:GvzBw6WI6KoQuRHRFoKcRvYGrBs=', 'Content-Length': '0'} 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000012d8399997fa87d9-0065e61417-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-77 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577239.970066_1090.1","last_update":"2024-03-04T18:33:59.970066Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:00.298+0000 7f7266d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:00.299+0000 7f7266d22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:00Z"} 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:00Z"} 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:05.427+0000 7fb407015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:05.428+0000 7fb407015800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:05Z"} 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:05Z"} 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:10.563+0000 7fe9df444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:10.564+0000 7fe9df444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:10Z"} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:10Z"} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:15.692+0000 7fb10af2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:15.692+0000 7fb10af2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:15Z"} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:15Z"} 2024-03-04T19:09:57.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:20.840+0000 7f5eb6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:20.841+0000 7f5eb6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:20Z"} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:20Z"} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:25.981+0000 7f8c6f530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:25.982+0000 7f8c6f530800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:25Z"} 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:25Z"} 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:31.129+0000 7f58ba12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:31.130+0000 7f58ba12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:31Z"} 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:31Z"} 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:36.270+0000 7f829ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:36.271+0000 7f829ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:36Z"} 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:36Z"} 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:41.555+0000 7efd7e62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:41.556+0000 7efd7e62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:41Z"} 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:41Z"} 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:46.694+0000 7f2b6d327800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:46.695+0000 7f2b6d327800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:46Z"} 2024-03-04T19:09:57.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:46Z"} 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:51.842+0000 7f876534b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:51.843+0000 7f876534b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:51Z"} 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:51Z"} 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:56.999+0000 7f644bc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:34:57.000+0000 7f644bc44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:57Z"} 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:34:57Z"} 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:02.135+0000 7f3d21412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:02.136+0000 7f3d21412800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:02Z"} 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:02Z"} 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:07.271+0000 7f1060a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:07.272+0000 7f1060a13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:07Z"} 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:07Z"} 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:12.421+0000 7fe461f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:12.422+0000 7fe461f2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:12Z"} 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:12Z"} 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:17.563+0000 7ff799e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:17.564+0000 7ff799e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:17Z"} 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:17Z"} 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:22.692+0000 7f0992413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:22.693+0000 7f0992413800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:22Z"} 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:22Z"} 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:27.827+0000 7f326cb42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:27.828+0000 7f326cb42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:27Z"} 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:27Z"} 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:32.983+0000 7f288f5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:32.984+0000 7f288f5a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:32Z"} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:32Z"} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.218376_725.1 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:38.117+0000 7f7c5ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:38.118+0000 7f7c5ce2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:38Z"} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:38Z"} 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:43.329+0000 7fc407e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:43.330+0000 7fc407e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:43Z"} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:43Z"} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:48.458+0000 7faaf4c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:48.459+0000 7faaf4c30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:48Z"} 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:48Z"} 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:53.591+0000 7f743472b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:53.592+0000 7f743472b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:53Z"} 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:53Z"} 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:58.720+0000 7f50d482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:35:58.720+0000 7f50d482e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:58Z"} 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:35:58Z"} 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:03.865+0000 7f6c74d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:03.866+0000 7f6c74d24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:03Z"} 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.159615_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.159615Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:03Z"} 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.159615_716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.159615Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.159615_716.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.159615_716.1 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:09.012+0000 7f7156f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:09.013+0000 7f7156f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:09Z"} 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:09Z"} 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:14.154+0000 7fead2c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:14.155+0000 7fead2c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:14Z"} 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:14Z"} 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:19.302+0000 7f751d014800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:19.303+0000 7f751d014800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:19Z"} 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:19Z"} 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:24.445+0000 7effbd62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:24.446+0000 7effbd62e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:24Z"} 2024-03-04T19:09:57.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:24Z"} 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.214996_723.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.214996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.214996_723.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.214996_723.1 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:29.703+0000 7fda52e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:29.707+0000 7fda52e2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:29Z"} 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:29Z"} 2024-03-04T19:09:57.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:34.862+0000 7f36ab733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:34.863+0000 7f36ab733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:34Z"} 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:34Z"} 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:40.017+0000 7f0aee7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:40.018+0000 7f0aee7a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:40Z"} 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:40Z"} 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:45.159+0000 7fcd9d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:45.160+0000 7fcd9d333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:45Z"} 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:45Z"} 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:50.303+0000 7fac96533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:50.304+0000 7fac96533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:50Z"} 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:50Z"} 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:55.429+0000 7f7cc0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:36:55.430+0000 7f7cc0fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:55Z"} 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:36:55Z"} 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:00.557+0000 7fc2bfa30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:00.558+0000 7fc2bfa30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:00Z"} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:00Z"} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.154969_713.1 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:05.682+0000 7fb993431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:05.683+0000 7fb993431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:05Z"} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:05Z"} 2024-03-04T19:09:57.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:10.824+0000 7f1f80c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:10.825+0000 7f1f80c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:10Z"} 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:10Z"} 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:15.966+0000 7f1ea253b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:15.967+0000 7f1ea253b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:15Z"} 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:15Z"} 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:21.110+0000 7f9188339800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:21.111+0000 7f9188339800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:21Z"} 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:21Z"} 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:26.263+0000 7f26db94b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:26.264+0000 7f26db94b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:26Z"} 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:26Z"} 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:31.463+0000 7f2f8cf22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:31.470+0000 7f2f8cf22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:31Z"} 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:31Z"} 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:36.697+0000 7fc822c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:36.697+0000 7fc822c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:36Z"} 2024-03-04T19:09:57.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:36Z"} 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:41.835+0000 7f37c868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:41.836+0000 7f37c868c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:41Z"} 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:41Z"} 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:46.978+0000 7f3d92d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:46.979+0000 7f3d92d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:46Z"} 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:46Z"} 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:52.112+0000 7f81cf20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:52.113+0000 7f81cf20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:52Z"} 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:52Z"} 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:57.239+0000 7fadb0b24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:37:57.240+0000 7fadb0b24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:57Z"} 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:37:57Z"} 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:02.374+0000 7fcb0a90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:02.375+0000 7fcb0a90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:02Z"} 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:02Z"} 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:07.531+0000 7f5d6dd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:07.532+0000 7f5d6dd31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:07Z"} 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:07Z"} 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:12.677+0000 7fbd18133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:12.678+0000 7fbd18133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:12Z"} 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:12Z"} 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:17.825+0000 7f04aa48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:17.826+0000 7f04aa48c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:17Z"} 2024-03-04T19:09:57.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:17Z"} 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:22.961+0000 7f6dd5e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:22.962+0000 7f6dd5e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:22Z"} 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:22Z"} 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:28.179+0000 7fe30120f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:28.180+0000 7fe30120f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:28Z"} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:28Z"} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:33.336+0000 7fce1e613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:33.337+0000 7fce1e613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:33Z"} 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:33Z"} 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:38.469+0000 7efe5c90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:38.470+0000 7efe5c90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:38Z"} 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:38Z"} 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:43.598+0000 7f497bf1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:43.599+0000 7f497bf1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:43Z"} 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:43Z"} 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:48.724+0000 7f1fe1526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:48.725+0000 7f1fe1526800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:48Z"} 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:48Z"} 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:53.859+0000 7fa40842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:53.860+0000 7fa40842e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:53Z"} 2024-03-04T19:09:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:53Z"} 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:58.993+0000 7f0ae0a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:38:58.994+0000 7f0ae0a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:58Z"} 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:38:58Z"} 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709571978.121915_709.1 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:04.123+0000 7f405ac1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:04.124+0000 7f405ac1b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:04Z"} 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:04Z"} 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577239.970066_1090.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577239.970066_1090.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 1934, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/sbnvju-78/ 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/sbnvju-78/ 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /sbnvju-78/ 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-03-04T19:09:57.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi027.front.sepia.ceph.com:8000 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:PUT 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:Mon, 04 Mar 2024 18:39:09 GMT 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:/sbnvju-78/ 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:AWS 4LUC3XJ67HIPQ3U1NKIY:d5itYgP7qYLTprx7D/c03hzsgpA= 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Mon, 04 Mar 2024 18:39:09 GMT', 'Authorization': 'AWS 4LUC3XJ67HIPQ3U1NKIY:d5itYgP7qYLTprx7D/c03hzsgpA=', 'Content-Length': '0'} 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007fa02eae5c329518-0065e6154d-5355-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Mon, 04 Mar 2024 18:39:09 GMT'), ('Connection', 'Keep-Alive')] 2024-03-04T19:09:57.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=sbnvju-78 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:09.500+0000 7ff97772f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:09.500+0000 7ff97772f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:09Z"} 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:09Z"} 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:14.642+0000 7fc36d330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:14.642+0000 7fc36d330800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:14Z"} 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:14Z"} 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:19.863+0000 7f1d4793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:19.866+0000 7f1d4793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:19Z"} 2024-03-04T19:09:57.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:19Z"} 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:25.033+0000 7f5cd829d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:25.034+0000 7f5cd829d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:25Z"} 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:25Z"} 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:30.176+0000 7f4387430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:30.177+0000 7f4387430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:30Z"} 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:30Z"} 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:35.300+0000 7fbe9052d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:35.301+0000 7fbe9052d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:35Z"} 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:35Z"} 2024-03-04T19:09:57.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:40.435+0000 7f6cc692f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:40.436+0000 7f6cc692f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:40Z"} 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:40Z"} 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:45.557+0000 7f9a61f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:45.558+0000 7f9a61f24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:45Z"} 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:45Z"} 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:50.690+0000 7fc9b31a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:50.702+0000 7fc9b31a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:50Z"} 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.175444_719.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.175444Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:50Z"} 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.175444_719.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.175444Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.175444_719.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.175444_719.1 2024-03-04T19:09:57.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:55.931+0000 7f8a05d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:39:55.932+0000 7f8a05d2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:55Z"} 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:39:55Z"} 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:01.056+0000 7f0c6d8c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:01.057+0000 7f0c6d8c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:01Z"} 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:01Z"} 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:06.193+0000 7f891a92a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:06.193+0000 7f891a92a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:06Z"} 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:06Z"} 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:11.315+0000 7f1b0608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:11.316+0000 7f1b0608c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:11Z"} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:11Z"} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:16.503+0000 7f2411e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:16.506+0000 7f2411e1e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:16Z"} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:16Z"} 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:21.781+0000 7f90bdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:21.787+0000 7f90bdf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:21Z"} 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:21Z"} 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:26.963+0000 7f3b4c613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:26.964+0000 7f3b4c613800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:26Z"} 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:26Z"} 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:32.109+0000 7fb917911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:32.110+0000 7fb917911800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.245429_730.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.245429Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:32Z"} 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.245429_730.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.245429Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:32Z"} 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.245429_730.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.245429Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.245429_730.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.245429_730.1 2024-03-04T19:09:57.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:37.252+0000 7fbb29e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:37.253+0000 7fbb29e39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:37Z"} 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:37Z"} 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:42.401+0000 7f19c6a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:42.402+0000 7f19c6a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:42Z"} 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.116657_707.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.116657Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:42Z"} 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.116657_707.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.116657Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.116657_707.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.116657_707.1 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:47.532+0000 7f9ced41b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:47.533+0000 7f9ced41b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:47Z"} 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:47Z"} 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:52.660+0000 7f781d612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:52.660+0000 7f781d612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:52Z"} 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:52Z"} 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:57.788+0000 7f951093c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:40:57.788+0000 7f951093c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:57Z"} 2024-03-04T19:09:57.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:40:57Z"} 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:02.937+0000 7fbaa3ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:02.938+0000 7fbaa3ba2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:02Z"} 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:02Z"} 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:08.092+0000 7f6a7f815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:08.093+0000 7f6a7f815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:08Z"} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:08Z"} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:13.234+0000 7f6b8cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:13.235+0000 7f6b8cf31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:13Z"} 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:13Z"} 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:18.373+0000 7f8b79a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:18.374+0000 7f8b79a16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:18Z"} 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:18Z"} 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:23.508+0000 7f5b43542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:23.509+0000 7f5b43542800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:23Z"} 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:23Z"} 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:28.662+0000 7f419cbaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:28.663+0000 7f419cbaf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:28Z"} 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:28Z"} 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:33.797+0000 7fcc14333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:33.798+0000 7fcc14333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:33Z"} 2024-03-04T19:09:57.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:33Z"} 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:39.033+0000 7fc652b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:39.034+0000 7fc652b31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:39Z"} 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:39Z"} 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:44.181+0000 7fd76849d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:44.182+0000 7fd76849d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:44Z"} 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:44Z"} 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:49.317+0000 7ff00aa24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:49.318+0000 7ff00aa24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:49Z"} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:49Z"} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:54.458+0000 7f0ecc436800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:54.459+0000 7f0ecc436800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:54Z"} 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:54Z"} 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:59.598+0000 7f7d9b3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:41:59.599+0000 7f7d9b3af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:59Z"} 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:41:59Z"} 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:04.744+0000 7f2d93a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:04.745+0000 7f2d93a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:04Z"} 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:04Z"} 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:09.893+0000 7ff118f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:09.894+0000 7ff118f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:09Z"} 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:09Z"} 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:15.044+0000 7f65adc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:15.045+0000 7f65adc24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:15Z"} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:15Z"} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:20.175+0000 7fd1f5d2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:20.176+0000 7fd1f5d2c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:20Z"} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:20Z"} 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:25.313+0000 7f82bbd2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:25.314+0000 7f82bbd2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:25Z"} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:25Z"} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:30.465+0000 7fd4fb73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:30.466+0000 7fd4fb73c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:30Z"} 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:30Z"} 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:35.655+0000 7f5cc3c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:35.656+0000 7f5cc3c39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:35Z"} 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:35Z"} 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:40.786+0000 7fba5bb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:40.787+0000 7fba5bb1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:40Z"} 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:40Z"} 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:45.911+0000 7f352a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:45.912+0000 7f352a933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:45Z"} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:45Z"} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:51.042+0000 7fa5f6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:51.043+0000 7fa5f6311800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:51Z"} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:51Z"} 2024-03-04T19:09:57.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:56.186+0000 7f1188a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:42:56.187+0000 7f1188a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:56Z"} 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:42:56Z"} 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:01.326+0000 7f1782e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:01.327+0000 7f1782e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:01Z"} 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:01Z"} 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:06.465+0000 7f541931e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:06.466+0000 7f541931e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:06Z"} 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:06Z"} 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:11.598+0000 7f3860b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:11.599+0000 7f3860b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:11Z"} 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:11Z"} 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:16.732+0000 7f138713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:16.733+0000 7f138713c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:16Z"} 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:16Z"} 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:21.879+0000 7f51b4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:21.879+0000 7f51b4733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:21Z"} 2024-03-04T19:09:57.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:21Z"} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:27.036+0000 7f530d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:27.037+0000 7f530d030800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:27Z"} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:27Z"} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:32.179+0000 7f855352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:32.180+0000 7f855352f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:32Z"} 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:32Z"} 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:37.319+0000 7f489d731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:37.320+0000 7f489d731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:37Z"} 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:37Z"} 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:42.548+0000 7fcaf869d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:42.551+0000 7fcaf869d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:42Z"} 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:42Z"} 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:47.737+0000 7f42f1d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:47.738+0000 7f42f1d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:47Z"} 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:47Z"} 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:52.869+0000 7f46a008c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:52.870+0000 7f46a008c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:52Z"} 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:52Z"} 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:58.005+0000 7f376734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:43:58.006+0000 7f376734b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:58Z"} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:43:58Z"} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:03.132+0000 7f8caa51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:03.133+0000 7f8caa51f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:03Z"} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:03Z"} 2024-03-04T19:09:57.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:08.274+0000 7f463ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:08.276+0000 7f463ca13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:08Z"} 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:08Z"} 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:13.402+0000 7f2dbf40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:13.403+0000 7f2dbf40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:13Z"} 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:13Z"} 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.526 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2124, in test_sync_policy_config_zonegroup 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:18.675+0000 7f3bac017800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:18.676+0000 7f3bac017800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:18Z"} 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:18Z"} 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:23.810+0000 7f9ee2a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:23.811+0000 7f9ee2a11800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:23Z"} 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:23Z"} 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:28.940+0000 7fadd3731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:28.941+0000 7fadd3731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:28Z"} 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:28Z"} 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:34.080+0000 7fd1f1024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:34.081+0000 7fd1f1024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:34Z"} 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:34Z"} 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.531 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:39.301+0000 7f8d12d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:39.302+0000 7f8d12d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:39Z"} 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:39Z"} 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:44.436+0000 7feed3017800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:44.437+0000 7feed3017800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:44Z"} 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:44Z"} 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:49.574+0000 7f8cdbe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:49.575+0000 7f8cdbe30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:49Z"} 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:49Z"} 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:54.698+0000 7fbb677a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:54.699+0000 7fbb677a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:54Z"} 2024-03-04T19:09:57.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:54Z"} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:59.849+0000 7f807e40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:44:59.850+0000 7f807e40f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:59Z"} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:44:59Z"} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:04.978+0000 7fb2f8f26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:04.979+0000 7fb2f8f26800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:04Z"} 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:04Z"} 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.154969_713.1 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:10.117+0000 7fb92e82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:10.118+0000 7fb92e82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:10Z"} 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:10Z"} 2024-03-04T19:09:57.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:15.261+0000 7f4169f22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:15.262+0000 7f4169f22800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:15Z"} 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:15Z"} 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:20.386+0000 7f9a36016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:20.387+0000 7f9a36016800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:20Z"} 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:20Z"} 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:25.536+0000 7f9dfeb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:25.537+0000 7f9dfeb2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:25Z"} 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:25Z"} 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:30.687+0000 7fef7b92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:30.688+0000 7fef7b92b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:30Z"} 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:30Z"} 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:35.841+0000 7f852df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:35.842+0000 7f852df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:35Z"} 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:35Z"} 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:40.988+0000 7fdd907a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:40.989+0000 7fdd907a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:40Z"} 2024-03-04T19:09:57.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:40Z"} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:46.201+0000 7f50aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:46.202+0000 7f50aa630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:46Z"} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:46Z"} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:51.362+0000 7f2d7de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:51.362+0000 7f2d7de30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:51Z"} 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:51Z"} 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:56.508+0000 7f72e61af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:45:56.508+0000 7f72e61af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:56Z"} 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:45:56Z"} 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:01.645+0000 7fc2c7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:01.646+0000 7fc2c7630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:01Z"} 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:01Z"} 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:06.772+0000 7ff29292d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:06.773+0000 7ff29292d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:06Z"} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:06Z"} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:11.915+0000 7f8bedd39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:11.916+0000 7f8bedd39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:11Z"} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:11Z"} 2024-03-04T19:09:57.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:17.063+0000 7f5e9731e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:17.064+0000 7f5e9731e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.714481_766.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.714481Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:17Z"} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709572290.714481_766.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:11:30.714481Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:17Z"} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709572290.714481_766.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:11:30.714481Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709572290.714481_766.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709572290.714481_766.1 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:22.193+0000 7f7397244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:22.193+0000 7f7397244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:22Z"} 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:22Z"} 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:27.326+0000 7f297ca12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:27.327+0000 7f297ca12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:27Z"} 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:27Z"} 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:32.453+0000 7f0ab89a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:32.453+0000 7f0ab89a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.383476_745.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.383476Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:32Z"} 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.383476_745.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.383476Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:32Z"} 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.383476_745.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.383476Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.383476_745.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.383476_745.1 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:37.592+0000 7f8e0422f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:37.593+0000 7f8e0422f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:37Z"} 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:37Z"} 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:42.736+0000 7f7570239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:42.737+0000 7f7570239800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:42Z"} 2024-03-04T19:09:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:42Z"} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:47.903+0000 7f7e9571f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:47.904+0000 7f7e9571f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:47Z"} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:47Z"} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:53.043+0000 7fa109b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:53.044+0000 7fa109b2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:53Z"} 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:53Z"} 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:58.249+0000 7f005d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:46:58.251+0000 7f005d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:58Z"} 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:46:58Z"} 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:03.533+0000 7f512793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:03.534+0000 7f512793b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:03Z"} 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:03Z"} 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:08.683+0000 7fc48cd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:08.684+0000 7fc48cd21800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:08Z"} 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:08Z"} 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:13.803+0000 7f42f8e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:13.804+0000 7f42f8e16800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:13Z"} 2024-03-04T19:09:57.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:13Z"} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:18.935+0000 7f3fdd529800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:18.936+0000 7f3fdd529800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:18Z"} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:18Z"} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:24.079+0000 7fcf7b7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:24.080+0000 7fcf7b7af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:24Z"} 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:24Z"} 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:29.212+0000 7fc82b212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:29.213+0000 7fc82b212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:29Z"} 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:29Z"} 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:34.343+0000 7f73a3a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:34.344+0000 7f73a3a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:34Z"} 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:34Z"} 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:39.476+0000 7f57c0c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:39.477+0000 7f57c0c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:39Z"} 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:39Z"} 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.153603_712.1 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:44.614+0000 7f6470e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:44.615+0000 7f6470e8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:44Z"} 2024-03-04T19:09:57.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:44Z"} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:49.744+0000 7f7670844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:49.745+0000 7f7670844800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.301088_735.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.301088Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:49Z"} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.301088_735.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.301088Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:49Z"} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.301088_735.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.301088Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.301088_735.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.301088_735.1 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:54.871+0000 7f845b00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:47:54.871+0000 7f845b00f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:54Z"} 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:47:54Z"} 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:00.013+0000 7fae2c30d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:00.014+0000 7fae2c30d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:00Z"} 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:00Z"} 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:05.177+0000 7f759ad2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:05.178+0000 7f759ad2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:05Z"} 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:05Z"} 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:10.311+0000 7f13e2533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:10.312+0000 7f13e2533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:10Z"} 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:10Z"} 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:15.442+0000 7fe0ac50b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:15.443+0000 7fe0ac50b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:15Z"} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:15Z"} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:20.582+0000 7f1ed511f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:20.583+0000 7f1ed511f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:20Z"} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:20Z"} 2024-03-04T19:09:57.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:25.713+0000 7f15a671f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:25.714+0000 7f15a671f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:25Z"} 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.121915_709.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.121915Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:25Z"} 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.121915_709.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.121915Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.121915_709.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.121915_709.1 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:30.847+0000 7f61c4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:30.848+0000 7f61c4830800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:30Z"} 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:30Z"} 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:36.051+0000 7f5bc7622800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:36.052+0000 7f5bc7622800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:36Z"} 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:36Z"} 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:41.312+0000 7f8ffa4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:41.313+0000 7f8ffa4c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:41Z"} 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:41Z"} 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:46.493+0000 7f9080d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:46.494+0000 7f9080d29800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:46Z"} 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:46Z"} 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:51.638+0000 7fef0b12c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:51.639+0000 7fef0b12c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:51Z"} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:51Z"} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:56.772+0000 7fc1757a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:48:56.773+0000 7fc1757a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:56Z"} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:48:56Z"} 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:01.911+0000 7fab94131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:01.911+0000 7fab94131800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:01Z"} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:01Z"} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:07.056+0000 7f1d25511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:07.057+0000 7f1d25511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:07Z"} 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:07Z"} 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:12.212+0000 7f49be533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:12.213+0000 7f49be533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:12Z"} 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:12Z"} 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:17.357+0000 7f13e0daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:17.358+0000 7f13e0daf800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:17Z"} 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:17Z"} 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:22.480+0000 7f28cc12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:22.481+0000 7f28cc12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:22Z"} 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:22Z"} 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-03-04T19:09:57.579 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2196, in test_sync_flow_symmetrical_zonegroup_all 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:27.763+0000 7fb42692d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:27.764+0000 7fb42692d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:27Z"} 2024-03-04T19:09:57.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:27Z"} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:32.905+0000 7f8d25f31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:32.906+0000 7f8d25f31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:32Z"} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:32Z"} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:38.142+0000 7fbddb02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:38.143+0000 7fbddb02e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:38Z"} 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:38Z"} 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:43.302+0000 7ff9c513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:43.303+0000 7ff9c513c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:43Z"} 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:43Z"} 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:48.461+0000 7f838aa34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:48.462+0000 7f838aa34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:48Z"} 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:48Z"} 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:53.603+0000 7f26d034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:53.604+0000 7f26d034b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:53Z"} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:53Z"} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:58.750+0000 7fce73814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:49:58.750+0000 7fce73814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:58Z"} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:49:58Z"} 2024-03-04T19:09:57.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:03.877+0000 7fd58a23a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:03.878+0000 7fd58a23a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:03Z"} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:03Z"} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:09.027+0000 7f99d012b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:09.028+0000 7f99d012b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:09Z"} 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:09Z"} 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:14.173+0000 7f9960a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:14.174+0000 7f9960a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:14Z"} 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:14Z"} 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:19.320+0000 7f0a7ef24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:19.321+0000 7f0a7ef24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:19Z"} 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:19Z"} 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:24.448+0000 7f8f70644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:24.449+0000 7f8f70644800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:24Z"} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:24Z"} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:29.591+0000 7f5edcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:29.592+0000 7f5edcd33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:29Z"} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:29Z"} 2024-03-04T19:09:57.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:34.713+0000 7fc900c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:34.714+0000 7fc900c12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:34Z"} 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:34Z"} 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:39.862+0000 7f0196415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:39.863+0000 7f0196415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:39Z"} 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:39Z"} 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:45.008+0000 7fa0e4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:45.008+0000 7fa0e4f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:45Z"} 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:45Z"} 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:50.238+0000 7fac3b612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:50.239+0000 7fac3b612800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:50Z"} 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:50Z"} 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:55.388+0000 7f2e0f20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:50:55.389+0000 7f2e0f20f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:55Z"} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:50:55Z"} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:00.519+0000 7ff664d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:00.520+0000 7ff664d33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:00Z"} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:00Z"} 2024-03-04T19:09:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:05.658+0000 7f3a36285800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:05.659+0000 7f3a36285800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:05Z"} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:05Z"} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:10.803+0000 7f32000c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:10.804+0000 7f32000c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:10Z"} 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:10Z"} 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:15.936+0000 7f0382331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:15.937+0000 7f0382331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:15Z"} 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:15Z"} 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:21.059+0000 7fe871711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:21.060+0000 7fe871711800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:21Z"} 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:21Z"} 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:26.194+0000 7fd10e442800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:26.195+0000 7fd10e442800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:26Z"} 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:26Z"} 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:31.333+0000 7f258ccc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:31.334+0000 7f258ccc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:31Z"} 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:31Z"} 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:36.477+0000 7efdd072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:36.478+0000 7efdd072f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:36Z"} 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:36Z"} 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:41.626+0000 7fca9f90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:41.627+0000 7fca9f90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.183700_722.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.183700Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:41Z"} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.183700_722.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.183700Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:41Z"} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.183700_722.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.183700Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.183700_722.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.183700_722.1 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:46.845+0000 7f108f811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:46.846+0000 7f108f811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:46Z"} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:46Z"} 2024-03-04T19:09:57.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:52.036+0000 7febf9f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:52.037+0000 7febf9f30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:52Z"} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:52Z"} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:57.172+0000 7feadab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:51:57.173+0000 7feadab1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:57Z"} 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:51:57Z"} 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:02.295+0000 7feabefa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:02.296+0000 7feabefa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.327181_739.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.327181Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:02Z"} 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.327181_739.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.327181Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:02Z"} 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.327181_739.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.327181Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.327181_739.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.327181_739.1 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:07.444+0000 7f7dd4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:07.445+0000 7f7dd4331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:07Z"} 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:07Z"} 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:12.568+0000 7fa513285800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:12.569+0000 7fa513285800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:12Z"} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:12Z"} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:17.700+0000 7fdd4e038800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:17.701+0000 7fdd4e038800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:17Z"} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:17Z"} 2024-03-04T19:09:57.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:22.836+0000 7ff62b528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:22.837+0000 7ff62b528800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:22Z"} 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:22Z"} 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:27.964+0000 7f673ac30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:27.965+0000 7f673ac30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:27Z"} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:27Z"} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:33.122+0000 7f58d4815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:33.123+0000 7f58d4815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:33Z"} 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:33Z"} 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:38.265+0000 7f24d8f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:38.266+0000 7f24d8f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:38Z"} 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:38Z"} 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:43.401+0000 7f3a15331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:43.402+0000 7f3a15331800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:43Z"} 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:43Z"} 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:48.583+0000 7fb60df2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:48.594+0000 7fb60df2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:48Z"} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:48Z"} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:53.787+0000 7f5cff92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:53.788+0000 7f5cff92f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.223764_728.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.223764Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:53Z"} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.223764_728.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.223764Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:53Z"} 2024-03-04T19:09:57.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.223764_728.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.223764Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.223764_728.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.223764_728.1 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:59.024+0000 7f4f0b824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:52:59.024+0000 7f4f0b824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:59Z"} 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:52:59Z"} 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:04.157+0000 7fae62614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:04.158+0000 7fae62614800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:04Z"} 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:04Z"} 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:09.305+0000 7f14eeb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:09.306+0000 7f14eeb2a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:09Z"} 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:09Z"} 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:14.429+0000 7f64c4f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:14.430+0000 7f64c4f33800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:14Z"} 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:14Z"} 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:19.562+0000 7f9c6a444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:19.563+0000 7f9c6a444800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:19Z"} 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:19Z"} 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:24.708+0000 7fb7cc511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:24.709+0000 7fb7cc511800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:24Z"} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:24Z"} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:29.855+0000 7fc08408c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:29.856+0000 7fc08408c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:29Z"} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:29Z"} 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:35.015+0000 7f4923f0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:35.016+0000 7f4923f0b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:35Z"} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:35Z"} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.242177_729.1 2024-03-04T19:09:57.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:40.173+0000 7fe2fc431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:40.174+0000 7fe2fc431800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:40Z"} 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:40Z"} 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:45.335+0000 7fa751e2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:45.336+0000 7fa751e2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:45Z"} 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:45Z"} 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:50.572+0000 7fe321333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:50.573+0000 7fe321333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:50Z"} 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:50Z"} 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:55.701+0000 7fdc10733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:53:55.702+0000 7fdc10733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:55Z"} 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:53:55Z"} 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:00.831+0000 7f3e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:00.832+0000 7f3e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:00Z"} 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:00Z"} 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:05.962+0000 7f854f812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:05.963+0000 7f854f812800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:05Z"} 2024-03-04T19:09:57.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:05Z"} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:11.109+0000 7ff225211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:11.110+0000 7ff225211800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:11Z"} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:11Z"} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:16.342+0000 7f72dfa3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:16.347+0000 7f72dfa3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:16Z"} 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:16Z"} 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:21.624+0000 7f925aa13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:21.630+0000 7f925aa13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:21Z"} 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:21Z"} 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:26.797+0000 7f6d44933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:26.798+0000 7f6d44933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:26Z"} 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:26Z"} 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:31.928+0000 7f555a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:31.929+0000 7f555a624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:31Z"} 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:31Z"} 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_select: 2024-03-04T19:09:57.632 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2259, in test_sync_flow_symmetrical_zonegroup_select 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:37.210+0000 7f50fe024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:37.211+0000 7f50fe024800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:37Z"} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:37Z"} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:42.346+0000 7fbd02a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:42.347+0000 7fbd02a20800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:42Z"} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:42Z"} 2024-03-04T19:09:57.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:47.484+0000 7f84a9333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:47.485+0000 7f84a9333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:47Z"} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:47Z"} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:52.637+0000 7f79bf013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:52.638+0000 7f79bf013800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:52Z"} 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:52Z"} 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:57.777+0000 7f2b4b333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:54:57.778+0000 7f2b4b333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:57Z"} 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:54:57Z"} 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:02.919+0000 7fd319c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:02.920+0000 7fd319c13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:02Z"} 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:02Z"} 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:08.075+0000 7f6b88044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:08.076+0000 7f6b88044800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:08Z"} 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:08Z"} 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:13.294+0000 7f0f30616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:13.295+0000 7f0f30616800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:13Z"} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:13Z"} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:18.426+0000 7ff0ed088800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:18.427+0000 7ff0ed088800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:18Z"} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:18Z"} 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:23.547+0000 7fd317e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:23.548+0000 7fd317e34800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:23Z"} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:23Z"} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.642 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:28.672+0000 7ff136c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:28.673+0000 7ff136c14800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:28Z"} 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:28Z"} 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:33.796+0000 7fa5ea88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:33.797+0000 7fa5ea88c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:33Z"} 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:33Z"} 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:38.929+0000 7f308aa44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:38.930+0000 7f308aa44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:38Z"} 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:38Z"} 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:44.066+0000 7f92e510f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:44.067+0000 7f92e510f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:44Z"} 2024-03-04T19:09:57.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.002120_706.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.002120Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:44Z"} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.002120_706.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.002120Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.002120_706.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.002120_706.1 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:49.209+0000 7f0d5aa0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:49.210+0000 7f0d5aa0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:49Z"} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:49Z"} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:54.345+0000 7f8490a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:54.345+0000 7f8490a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:54Z"} 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:54Z"} 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:59.584+0000 7fac4b424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:55:59.585+0000 7fac4b424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:59Z"} 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:55:59Z"} 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:04.712+0000 7f5a0982e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:04.712+0000 7f5a0982e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:04Z"} 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:04Z"} 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:09.850+0000 7fe9917a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:09.851+0000 7fe9917a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:09Z"} 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:09Z"} 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:15.000+0000 7f1224fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:15.001+0000 7f1224fa2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:15Z"} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:15Z"} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:20.121+0000 7f11eb42f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:20.122+0000 7f11eb42f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:20Z"} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:20Z"} 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:25.254+0000 7f580da2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:25.255+0000 7f580da2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:25Z"} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:25Z"} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:30.379+0000 7fe85bb3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:30.379+0000 7fe85bb3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:30Z"} 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:30Z"} 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:35.521+0000 7f7019c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:35.522+0000 7f7019c2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:35Z"} 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:35Z"} 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:40.660+0000 7fecd3111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:40.661+0000 7fecd3111800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:40Z"} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.998361_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.998361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:40Z"} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.998361_705.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.998361Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.998361_705.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.998361_705.1 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:45.792+0000 7f3ac1638800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:45.793+0000 7f3ac1638800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:45Z"} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:45Z"} 2024-03-04T19:09:57.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:50.932+0000 7f37f5810800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:50.933+0000 7f37f5810800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:50Z"} 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:50Z"} 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T19:09:57.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:56.067+0000 7f4efd067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:56:56.068+0000 7f4efd067800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:56Z"} 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:56:56Z"} 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:01.212+0000 7f755c615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:01.213+0000 7f755c615800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:01Z"} 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:01Z"} 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:06.355+0000 7fc767630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:06.356+0000 7fc767630800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:06Z"} 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:06Z"} 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:11.592+0000 7fec9a1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:11.592+0000 7fec9a1a2800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:11Z"} 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:11Z"} 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:16.719+0000 7f379df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:16.720+0000 7f379df31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:16Z"} 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:16Z"} 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:21.843+0000 7fe3fe214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:21.843+0000 7fe3fe214800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:21Z"} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:21Z"} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:26.974+0000 7efea3617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:26.975+0000 7efea3617800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:26Z"} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:26Z"} 2024-03-04T19:09:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:32.121+0000 7f56f2e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:32.122+0000 7f56f2e30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:32Z"} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:32Z"} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:37.282+0000 7f5a6e624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:37.283+0000 7f5a6e624800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:37Z"} 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:37Z"} 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:42.524+0000 7efefe133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:42.525+0000 7efefe133800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:42Z"} 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:42Z"} 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:47.689+0000 7f3c4a333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:47.690+0000 7f3c4a333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:47Z"} 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.214996_723.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.214996Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:47Z"} 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.214996_723.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.214996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.214996_723.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.214996_723.1 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:52.821+0000 7f0e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:52.822+0000 7f0e20424800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:52Z"} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:52Z"} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:57.965+0000 7fa29b733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:57:57.966+0000 7fa29b733800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:57Z"} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:57:57Z"} 2024-03-04T19:09:57.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:03.096+0000 7fe9f1467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:03.097+0000 7fe9f1467800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:03Z"} 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:03Z"} 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:08.222+0000 7f32fa415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:08.223+0000 7f32fa415800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:08Z"} 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.125073_710.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.125073Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:08Z"} 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.125073_710.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.125073Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.125073_710.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.125073_710.1 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:13.365+0000 7fe9fd82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:13.366+0000 7fe9fd82e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:13Z"} 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:13Z"} 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:18.508+0000 7f05b861b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:18.509+0000 7f05b861b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:18Z"} 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:18Z"} 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:23.643+0000 7fcfe333c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:23.644+0000 7fcfe333c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:23Z"} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:23Z"} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:28.794+0000 7f112e414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:28.795+0000 7f112e414800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:28Z"} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:28Z"} 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:33.950+0000 7f7891a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:33.951+0000 7f7891a30800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:33Z"} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:33Z"} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:39.107+0000 7fa78652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:39.108+0000 7fa78652f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:39Z"} 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:39Z"} 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:44.256+0000 7f4200333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:44.256+0000 7f4200333800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571979.256260_731.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:19.256260Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:44Z"} 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571979.256260_731.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:19.256260Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:44Z"} 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571979.256260_731.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:19.256260Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571979.256260_731.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571979.256260_731.1 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:49.506+0000 7fc690815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:49.509+0000 7fc690815800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:49Z"} 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:49Z"} 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:54.645+0000 7f88a9212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:54.645+0000 7f88a9212800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:54Z"} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:54Z"} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:59.794+0000 7f6b1ce13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:58:59.794+0000 7f6b1ce13800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:59Z"} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:58:59Z"} 2024-03-04T19:09:57.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:04.945+0000 7f7e4662d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:04.946+0000 7f7e4662d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:04Z"} 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.886755_642.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.886755Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:04Z"} 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.886755_642.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.886755Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.886755_642.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.886755_642.1 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:10.074+0000 7fb73bc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:10.075+0000 7fb73bc8c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:10Z"} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:10Z"} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:15.213+0000 7efee2213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:15.214+0000 7efee2213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:15Z"} 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:15Z"} 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:20.352+0000 7f2c09130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:20.353+0000 7f2c09130800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:20Z"} 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:20Z"} 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:25.496+0000 7f02df32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:25.497+0000 7f02df32a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:25Z"} 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:25Z"} 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:30.634+0000 7f91d428a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:30.635+0000 7f91d428a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:30Z"} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:30Z"} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:35.772+0000 7ff45013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:35.774+0000 7ff45013b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:35Z"} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:35Z"} 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:40.918+0000 7f2c9f244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:40.919+0000 7f2c9f244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:40Z"} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:40Z"} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-03-04T19:09:57.687 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 2329, in test_sync_flow_directional_zonegroup_select 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 210, in zonegroup_meta_checkpoint 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/../src/test/rgw/rgw_multi/tests.py", line 199, in zone_meta_checkpoint 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stderr: INFO: No --period given, using current period=0c9eafce-04d6-4013-9950-f1b02bed5b7e 2024-03-04T19:09:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi027.stdout: INFO: [{"marker":"1_1709577549.182425_1108.1","last_update":"2024-03-04T18:39:09.182425Z"},{"marker":"1_1709576003.291343_1334.1","last_update":"2024-03-04T18:13:23.291343Z"},{"marker":"1_1709575694.049259_2454.1","last_update":"2024-03-04T18:08:14.049259Z"},{"marker":"1_1709575694.071129_358.1","last_update":"2024-03-04T18:08:14.071129Z"}] 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:46.707+0000 7fd63ff31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:46.708+0000 7fd63ff31800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:46Z"} 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:46Z"} 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:57.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:51.846+0000 7fc237f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:51.847+0000 7fc237f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:51Z"} 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:51Z"} 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:56.980+0000 7f8c153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T18:59:56.981+0000 7f8c153af800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:56Z"} 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T18:59:56Z"} 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:02.112+0000 7f1a4c80f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:02.113+0000 7f1a4c80f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:02Z"} 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.242177_729.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.242177Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:02Z"} 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.242177_729.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.242177Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.242177_729.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.242177_729.1 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:07.251+0000 7f19fac24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:07.252+0000 7f19fac24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:07Z"} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:07Z"} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:12.483+0000 7f85d0a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:12.484+0000 7f85d0a24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:12Z"} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:12Z"} 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:17.622+0000 7fe91509d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:17.623+0000 7fe91509d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:17Z"} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:17Z"} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:22.765+0000 7f67c6f0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:22.766+0000 7f67c6f0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:22Z"} 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:22Z"} 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:27.917+0000 7f2c52230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:27.918+0000 7f2c52230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:27Z"} 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:27Z"} 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:33.050+0000 7f49bff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:33.051+0000 7f49bff2d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:33Z"} 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.153603_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.153603Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:33Z"} 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.153603_712.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.153603Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.153603_712.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.153603_712.1 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:38.185+0000 7f6a9908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:38.186+0000 7f6a9908c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.351182_743.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.351182Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:38Z"} 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.351182_743.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.351182Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:38Z"} 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.351182_743.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.351182Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.351182_743.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.351182_743.1 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:43.324+0000 7f6605d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:43.325+0000 7f6605d42800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:43Z"} 2024-03-04T19:09:57.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:43Z"} 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:48.459+0000 7f1511a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:48.460+0000 7f1511a2e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:48Z"} 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:48Z"} 2024-03-04T19:09:57.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:53.596+0000 7f969ad24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:53.597+0000 7f969ad24800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:53Z"} 2024-03-04T19:09:57.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:53Z"} 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:58.746+0000 7f72d0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:00:58.748+0000 7f72d0a83800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:58Z"} 2024-03-04T19:09:57.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:00:58Z"} 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:03.980+0000 7f69d2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:03.983+0000 7f69d2d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:03Z"} 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:03Z"} 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:09.169+0000 7f7f97f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:09.169+0000 7f7f97f39800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:09Z"} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:09Z"} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:14.296+0000 7f54ec12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:14.297+0000 7f54ec12f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:14Z"} 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.842248_699.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.842248Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:14Z"} 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.842248_699.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.842248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.842248_699.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.842248_699.1 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:19.419+0000 7f74d9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:19.420+0000 7f74d9933800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:19Z"} 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:19Z"} 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:24.550+0000 7f2256731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:24.551+0000 7f2256731800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:24Z"} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.120400_708.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.120400Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:24Z"} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.120400_708.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.120400Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.120400_708.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.120400_708.1 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:29.675+0000 7f7e7e89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:29.676+0000 7f7e7e89d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:29Z"} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:29Z"} 2024-03-04T19:09:57.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:34.802+0000 7f5ab0a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:34.803+0000 7f5ab0a3a800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:34Z"} 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:34Z"} 2024-03-04T19:09:57.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:39.942+0000 7f6249f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:39.943+0000 7f6249f4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:39Z"} 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.218376_725.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.218376Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:39Z"} 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.218376_725.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.218376Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.218376_725.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.218376_725.1 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:45.065+0000 7f64d7730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:45.066+0000 7f64d7730800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:45Z"} 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:45Z"} 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:50.197+0000 7f451402d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:50.198+0000 7f451402d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:50Z"} 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:50Z"} 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:55.330+0000 7f8c0042e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:01:55.331+0000 7f8c0042e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:55Z"} 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.154969_713.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.154969Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:01:55Z"} 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.154969_713.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.154969Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.154969_713.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.154969_713.1 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:00.459+0000 7f13c109d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:00.460+0000 7f13c109d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:00Z"} 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:00Z"} 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:05.595+0000 7f48e360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:05.596+0000 7f48e360f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:05Z"} 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:05Z"} 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:10.731+0000 7f75fb90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:10.732+0000 7f75fb90d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:10Z"} 2024-03-04T19:09:57.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:10Z"} 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:16.030+0000 7f23e7c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:16.031+0000 7f23e7c9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:16Z"} 2024-03-04T19:09:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.150325_711.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.150325Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:16Z"} 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.150325_711.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.150325Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.150325_711.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.150325_711.1 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:21.187+0000 7fdfaf2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:21.188+0000 7fdfaf2c0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:21Z"} 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:21Z"} 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:26.323+0000 7f7cf4814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:26.324+0000 7f7cf4814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:26Z"} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.996460_704.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.996460Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:26Z"} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.996460_704.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.996460Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.996460_704.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.996460_704.1 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:31.463+0000 7ff147533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:31.464+0000 7ff147533800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:31Z"} 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.684679_567.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.684679Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:31Z"} 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.684679_567.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.684679Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.684679_567.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.684679_567.1 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:36.602+0000 7f3fad236800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:36.602+0000 7f3fad236800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:36Z"} 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:36Z"} 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:41.740+0000 7f7ba4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:41.741+0000 7f7ba4d3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:41Z"} 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:41Z"} 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:46.886+0000 7fa1a0244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:46.887+0000 7fa1a0244800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:46Z"} 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:46Z"} 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:52.028+0000 7fcc94b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:52.029+0000 7fcc94b1f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:52Z"} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:52Z"} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:57.164+0000 7f813d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:02:57.165+0000 7f813d213800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:57Z"} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:02:57Z"} 2024-03-04T19:09:57.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:02.298+0000 7f33e3a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:02.299+0000 7f33e3a0f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:02Z"} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:02Z"} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:07.439+0000 7fd51f937800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:07.440+0000 7fd51f937800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:07Z"} 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:07Z"} 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:12.646+0000 7fcb24c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:12.646+0000 7fcb24c44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:12Z"} 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:12Z"} 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:17.819+0000 7f3979a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:17.820+0000 7f3979a44800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:17Z"} 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.890340_643.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.890340Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:17Z"} 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.890340_643.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.890340Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.890340_643.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.890340_643.1 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:22.947+0000 7fcc4acc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:22.948+0000 7fcc4acc0800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:22Z"} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.644921_566.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.644921Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:22Z"} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.644921_566.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.644921Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.644921_566.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.644921_566.1 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:28.082+0000 7fbdf614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:28.083+0000 7fbdf614b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:28Z"} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:28Z"} 2024-03-04T19:09:57.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:33.227+0000 7f2dcec12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:33.228+0000 7f2dcec12800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:33Z"} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.885240_641.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.885240Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:33Z"} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.885240_641.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.885240Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.885240_641.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.885240_641.1 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:38.364+0000 7f6f00824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:38.365+0000 7f6f00824800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:38Z"} 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.641846_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.641846Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:38Z"} 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.641846_565.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.641846Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.641846_565.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.641846_565.1 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:43.509+0000 7f518b11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:43.510+0000 7f518b11f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:43Z"} 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:43Z"} 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:48.654+0000 7f7627814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:48.655+0000 7f7627814800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:48Z"} 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571978.220171_727.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:18.220171Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:48Z"} 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571978.220171_727.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:18.220171Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571978.220171_727.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571978.220171_727.1 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:53.807+0000 7f32fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:53.808+0000 7f32fb410800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:53Z"} 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.828757_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.828757Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:53Z"} 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.828757_695.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.828757Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.828757_695.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.828757_695.1 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:58.948+0000 7fc32bf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:03:58.949+0000 7fc32bf2f800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:58Z"} 2024-03-04T19:09:57.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.835525_697.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.835525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:03:58Z"} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.835525_697.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.835525Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.835525_697.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.835525_697.1 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:04.227+0000 7f86d651e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:04.227+0000 7f86d651e800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:04Z"} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571980.296697_734.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:20.296697Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:04Z"} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571980.296697_734.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:20.296697Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571980.296697_734.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571980.296697_734.1 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:09.353+0000 7f05d5034800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:09.353+0000 7f05d5034800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:09Z"} 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:09Z"} 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:14.490+0000 7fee00430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:14.491+0000 7fee00430800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:14Z"} 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.688202_568.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.688202Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:14Z"} 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.688202_568.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.688202Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.688202_568.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.688202_568.1 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:19.612+0000 7f547ec9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:19.613+0000 7f547ec9d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:19Z"} 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.640587_564.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.640587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:19Z"} 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.640587_564.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.640587Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.640587_564.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.640587_564.1 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:24.740+0000 7fb8d7811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:24.740+0000 7fb8d7811800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:24Z"} 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:24Z"} 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:29.879+0000 7f056a230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:29.880+0000 7f056a230800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:29Z"} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.692350_570.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.692350Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:29Z"} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.692350_570.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.692350Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.692350_570.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.692350_570.1 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:35.012+0000 7f72e4b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:35.013+0000 7f72e4b4b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:35Z"} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.987908_703.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.987908Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:35Z"} 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.987908_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.987908Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.987908_703.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.987908_703.1 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:40.145+0000 7f5f6002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:40.145+0000 7f5f6002d800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:40Z"} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571357.881246_635.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:55:57.881246Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:40Z"} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571357.881246_635.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:55:57.881246Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571357.881246_635.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571357.881246_635.1 2024-03-04T19:09:57.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:45.281+0000 7ffbf0f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:45.282+0000 7ffbf0f3c800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:45Z"} 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709570988.689443_569.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T16:49:48.689443Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:45Z"} 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709570988.689443_569.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T16:49:48.689443Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709570988.689443_569.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709570988.689443_569.1 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-03-04T19:09:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setuser ceph since I am not root 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:50.417+0000 7f4fa0f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stderr: INFO: 2024-03-04T19:04:50.418+0000 7f4fa0f2b800 20 RGW-SYNC:meta: read sync status 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi071.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:50Z"} 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0c9eafce-04d6-4013-9950-f1b02bed5b7e","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1709571977.838159_698.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-03-04T17:06:17.838159Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1709576003.291343_1334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:13:23.291343Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1709575694.049259_2454.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.049259Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1709575694.071129_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-03-04T18:08:14.071129Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-03-04T19:04:50Z"} 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1709571977.838159_698.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-03-04T17:06:17.838159Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1709576003.291343_1334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:13:23.291343Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1709575694.049259_2454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.049259Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1709575694.071129_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-03-04T18:08:14.071129Z', 'realm_epoch': 2}}] 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1709577549.182425_1108.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1709571977.838159_698.1', 1: '1_1709576003.291343_1334.1', 2: '1_1709575694.049259_2454.1', 3: '1_1709575694.071129_358.1'} 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=0 master=1_1709577549.182425_1108.1 target=1_1709571977.838159_698.1 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.740 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-03-04T19:09:57.741 INFO:tasks.rgw_multisite_tests:Ran 63 tests in 10637.976s 2024-03-04T19:09:57.741 INFO:tasks.rgw_multisite_tests: 2024-03-04T19:09:57.741 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=15, errors=4, failures=29) 2024-03-04T19:09:57.746 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-03-04T19:09:58.041 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=14828d6955c7492784b96dc5a3afd0e5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-03-04T19:09:58.042 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-03-04T19:09:58.054 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-03-04T19:09:58.067 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-03-04T19:09:58.078 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-03-04T19:09:58.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:09:58.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:09:58.080 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:09:58.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T19:09:58.097 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-03-04T19:09:58.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:09:58.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~151s 2024-03-04T19:09:58.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~151s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~150s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~150s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~150s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~150s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~150s 2024-03-04T19:09:58.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~150s 2024-03-04T19:09:58.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~150s 2024-03-04T19:10:01.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:03.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~156s 2024-03-04T19:10:03.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~156s 2024-03-04T19:10:03.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~156s 2024-03-04T19:10:03.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~156s 2024-03-04T19:10:03.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~156s 2024-03-04T19:10:03.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~156s 2024-03-04T19:10:03.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~156s 2024-03-04T19:10:03.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~155s 2024-03-04T19:10:03.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~155s 2024-03-04T19:10:08.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~161s 2024-03-04T19:10:08.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~161s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~160s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~160s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~160s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~160s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~160s 2024-03-04T19:10:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~160s 2024-03-04T19:10:08.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~160s 2024-03-04T19:10:10.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:10.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:13.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~166s 2024-03-04T19:10:13.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~166s 2024-03-04T19:10:13.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~165s 2024-03-04T19:10:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~165s 2024-03-04T19:10:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~165s 2024-03-04T19:10:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~165s 2024-03-04T19:10:13.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~165s 2024-03-04T19:10:13.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~165s 2024-03-04T19:10:13.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~165s 2024-03-04T19:10:18.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~171s 2024-03-04T19:10:18.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~171s 2024-03-04T19:10:18.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~171s 2024-03-04T19:10:18.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~171s 2024-03-04T19:10:18.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~171s 2024-03-04T19:10:18.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~171s 2024-03-04T19:10:18.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~171s 2024-03-04T19:10:18.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~170s 2024-03-04T19:10:18.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~170s 2024-03-04T19:10:18.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~170s 2024-03-04T19:10:18.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~170s 2024-03-04T19:10:18.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~170s 2024-03-04T19:10:18.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~170s 2024-03-04T19:10:18.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~170s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~176s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~176s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~176s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~176s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~176s 2024-03-04T19:10:23.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~176s 2024-03-04T19:10:23.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~176s 2024-03-04T19:10:23.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~175s 2024-03-04T19:10:23.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~175s 2024-03-04T19:10:23.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~175s 2024-03-04T19:10:23.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~175s 2024-03-04T19:10:23.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~175s 2024-03-04T19:10:23.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~175s 2024-03-04T19:10:23.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~175s 2024-03-04T19:10:25.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:25.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~181s 2024-03-04T19:10:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~181s 2024-03-04T19:10:28.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~181s 2024-03-04T19:10:28.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~181s 2024-03-04T19:10:28.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~181s 2024-03-04T19:10:28.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~181s 2024-03-04T19:10:28.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~181s 2024-03-04T19:10:28.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:28.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~180s 2024-03-04T19:10:28.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~180s 2024-03-04T19:10:28.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~180s 2024-03-04T19:10:28.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~180s 2024-03-04T19:10:28.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~180s 2024-03-04T19:10:28.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~180s 2024-03-04T19:10:28.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~180s 2024-03-04T19:10:31.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:31.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:31.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:31.388 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:10:31.388 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 2024-03-04T19:10:31.403 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T19:10:31.403 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T19:10:31.404 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-03-04T19:10:31.404 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:10:31.407 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:10:31.407 ERROR:teuthology.misc:Saw exception from c2.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0' 2024-03-04T19:10:31.424 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-03-04T19:10:31.425 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:10:31.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:10:31.425 ERROR:teuthology.misc:Saw exception from c2.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1' 2024-03-04T19:10:31.439 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-03-04T19:10:31.439 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:10:31.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:10:31.440 ERROR:teuthology.misc:Saw exception from c2.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' 2024-03-04T19:10:31.455 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T19:10:31.455 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-03-04T19:10:31.455 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:10:31.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:10:31.456 ERROR:teuthology.misc:Saw exception from c2.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' 2024-03-04T19:10:31.474 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T19:10:31.475 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-03-04T19:10:31.475 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:10:31.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:10:31.475 ERROR:teuthology.misc:Saw exception from c2.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi071 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a' 2024-03-04T19:10:31.497 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T19:10:31.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:31.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:33.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~186s 2024-03-04T19:10:33.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~186s 2024-03-04T19:10:33.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2024-03-04T19:10:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~185s 2024-03-04T19:10:34.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:38.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~191s 2024-03-04T19:10:38.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~191s 2024-03-04T19:10:38.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~190s 2024-03-04T19:10:38.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~190s 2024-03-04T19:10:38.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~190s 2024-03-04T19:10:38.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~190s 2024-03-04T19:10:38.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~190s 2024-03-04T19:10:38.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~190s 2024-03-04T19:10:38.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~190s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~196s 2024-03-04T19:10:43.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~196s 2024-03-04T19:10:43.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:43.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:43.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~195s 2024-03-04T19:10:43.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~195s 2024-03-04T19:10:43.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~195s 2024-03-04T19:10:43.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~195s 2024-03-04T19:10:43.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~195s 2024-03-04T19:10:43.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~195s 2024-03-04T19:10:43.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~195s 2024-03-04T19:10:46.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:10:48.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~201s 2024-03-04T19:10:48.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~201s 2024-03-04T19:10:48.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~201s 2024-03-04T19:10:48.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~201s 2024-03-04T19:10:48.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~201s 2024-03-04T19:10:48.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~201s 2024-03-04T19:10:48.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~201s 2024-03-04T19:10:48.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~200s 2024-03-04T19:10:48.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~200s 2024-03-04T19:10:48.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~200s 2024-03-04T19:10:48.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~200s 2024-03-04T19:10:48.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~200s 2024-03-04T19:10:48.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~200s 2024-03-04T19:10:48.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~200s 2024-03-04T19:10:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~206s 2024-03-04T19:10:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~206s 2024-03-04T19:10:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~206s 2024-03-04T19:10:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~206s 2024-03-04T19:10:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~206s 2024-03-04T19:10:53.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~206s 2024-03-04T19:10:53.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~206s 2024-03-04T19:10:53.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~205s 2024-03-04T19:10:53.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~205s 2024-03-04T19:10:53.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~205s 2024-03-04T19:10:53.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~205s 2024-03-04T19:10:53.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~205s 2024-03-04T19:10:53.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~205s 2024-03-04T19:10:53.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~205s 2024-03-04T19:10:58.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~211s 2024-03-04T19:10:58.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~211s 2024-03-04T19:10:58.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~211s 2024-03-04T19:10:58.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~211s 2024-03-04T19:10:58.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~211s 2024-03-04T19:10:58.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~211s 2024-03-04T19:10:58.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~211s 2024-03-04T19:10:58.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:10:58.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:10:58.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~210s 2024-03-04T19:10:58.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~210s 2024-03-04T19:11:01.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:11:03.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~216s 2024-03-04T19:11:03.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~216s 2024-03-04T19:11:03.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~215s 2024-03-04T19:11:03.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~215s 2024-03-04T19:11:03.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~215s 2024-03-04T19:11:03.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~215s 2024-03-04T19:11:03.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~215s 2024-03-04T19:11:03.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~215s 2024-03-04T19:11:03.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~215s 2024-03-04T19:11:04.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-04T19:11:04.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:04.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-04T19:11:04.764 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:11:04.764 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1144, in cluster if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1141, in first_in_ceph_log stdout = mon0_remote.sh(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi071 2024-03-04T19:11:04.785 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T19:11:04.795 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-03-04T19:11:04.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:04.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:07.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:08.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~221s 2024-03-04T19:11:08.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~221s 2024-03-04T19:11:08.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~221s 2024-03-04T19:11:08.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~221s 2024-03-04T19:11:08.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~221s 2024-03-04T19:11:08.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~221s 2024-03-04T19:11:08.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~221s 2024-03-04T19:11:08.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~220s 2024-03-04T19:11:08.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~220s 2024-03-04T19:11:13.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~226s 2024-03-04T19:11:13.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~226s 2024-03-04T19:11:13.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~225s 2024-03-04T19:11:13.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~225s 2024-03-04T19:11:16.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:16.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~231s 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~231s 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~231s 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~231s 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~231s 2024-03-04T19:11:18.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~231s 2024-03-04T19:11:18.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~231s 2024-03-04T19:11:18.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~230s 2024-03-04T19:11:18.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~230s 2024-03-04T19:11:18.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~230s 2024-03-04T19:11:18.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~230s 2024-03-04T19:11:18.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2024-03-04T19:11:18.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2024-03-04T19:11:18.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~230s 2024-03-04T19:11:19.932 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:23.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~236s 2024-03-04T19:11:23.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~236s 2024-03-04T19:11:23.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~236s 2024-03-04T19:11:23.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~236s 2024-03-04T19:11:23.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~236s 2024-03-04T19:11:23.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~236s 2024-03-04T19:11:23.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~236s 2024-03-04T19:11:23.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2024-03-04T19:11:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~235s 2024-03-04T19:11:28.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~241s 2024-03-04T19:11:28.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~241s 2024-03-04T19:11:28.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~241s 2024-03-04T19:11:28.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~241s 2024-03-04T19:11:28.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2024-03-04T19:11:28.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~241s 2024-03-04T19:11:28.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~241s 2024-03-04T19:11:28.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~240s 2024-03-04T19:11:28.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~240s 2024-03-04T19:11:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~240s 2024-03-04T19:11:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~240s 2024-03-04T19:11:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~240s 2024-03-04T19:11:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~240s 2024-03-04T19:11:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~240s 2024-03-04T19:11:31.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:31.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~246s 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~246s 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~246s 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~246s 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~246s 2024-03-04T19:11:33.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~246s 2024-03-04T19:11:33.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~246s 2024-03-04T19:11:33.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~245s 2024-03-04T19:11:33.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~245s 2024-03-04T19:11:33.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~245s 2024-03-04T19:11:33.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~245s 2024-03-04T19:11:33.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~245s 2024-03-04T19:11:33.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~245s 2024-03-04T19:11:33.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~245s 2024-03-04T19:11:35.004 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:38.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:38.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:38.076 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:38.076 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:11:38.076 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:11:38.092 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T19:11:38.092 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T19:11:38.093 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-03-04T19:11:38.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:11:38.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:11:38.096 ERROR:teuthology.misc:Saw exception from c1.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0' 2024-03-04T19:11:38.111 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-03-04T19:11:38.111 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:11:38.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:11:38.111 ERROR:teuthology.misc:Saw exception from c1.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1' 2024-03-04T19:11:38.126 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-03-04T19:11:38.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:11:38.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:11:38.127 ERROR:teuthology.misc:Saw exception from c1.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' 2024-03-04T19:11:38.142 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T19:11:38.143 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-03-04T19:11:38.143 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:11:38.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:11:38.143 ERROR:teuthology.misc:Saw exception from c1.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' 2024-03-04T19:11:38.161 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T19:11:38.161 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-03-04T19:11:38.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T19:11:38.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T19:11:38.161 ERROR:teuthology.misc:Saw exception from c1.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a' 2024-03-04T19:11:38.184 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T19:11:38.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:38.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:38.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~251s 2024-03-04T19:11:38.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~251s 2024-03-04T19:11:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~251s 2024-03-04T19:11:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~251s 2024-03-04T19:11:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~251s 2024-03-04T19:11:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~251s 2024-03-04T19:11:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~251s 2024-03-04T19:11:38.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~250s 2024-03-04T19:11:38.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~250s 2024-03-04T19:11:38.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~250s 2024-03-04T19:11:38.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~250s 2024-03-04T19:11:38.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~250s 2024-03-04T19:11:38.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~250s 2024-03-04T19:11:38.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~250s 2024-03-04T19:11:41.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:43.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~256s 2024-03-04T19:11:43.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~256s 2024-03-04T19:11:43.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~256s 2024-03-04T19:11:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~256s 2024-03-04T19:11:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~256s 2024-03-04T19:11:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~256s 2024-03-04T19:11:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~256s 2024-03-04T19:11:43.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~255s 2024-03-04T19:11:43.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~255s 2024-03-04T19:11:48.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~261s 2024-03-04T19:11:48.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~261s 2024-03-04T19:11:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~260s 2024-03-04T19:11:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~260s 2024-03-04T19:11:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~260s 2024-03-04T19:11:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~260s 2024-03-04T19:11:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~260s 2024-03-04T19:11:48.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~260s 2024-03-04T19:11:48.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~260s 2024-03-04T19:11:50.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:11:50.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:11:53.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~266s 2024-03-04T19:11:53.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~266s 2024-03-04T19:11:53.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~266s 2024-03-04T19:11:53.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~266s 2024-03-04T19:11:53.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~266s 2024-03-04T19:11:53.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~266s 2024-03-04T19:11:53.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~266s 2024-03-04T19:11:53.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:11:53.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~265s 2024-03-04T19:11:53.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~265s 2024-03-04T19:11:53.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~265s 2024-03-04T19:11:53.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~265s 2024-03-04T19:11:53.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~265s 2024-03-04T19:11:53.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~265s 2024-03-04T19:11:53.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~265s 2024-03-04T19:11:58.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~271s 2024-03-04T19:11:58.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~271s 2024-03-04T19:11:58.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~271s 2024-03-04T19:11:58.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~271s 2024-03-04T19:11:58.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~271s 2024-03-04T19:11:58.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~271s 2024-03-04T19:11:58.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~271s 2024-03-04T19:11:58.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~270s 2024-03-04T19:11:58.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~270s 2024-03-04T19:11:58.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~270s 2024-03-04T19:11:58.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~270s 2024-03-04T19:11:58.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~270s 2024-03-04T19:11:58.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~270s 2024-03-04T19:11:58.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~270s 2024-03-04T19:12:03.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~276s 2024-03-04T19:12:03.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~276s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2024-03-04T19:12:03.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~275s 2024-03-04T19:12:05.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:05.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:08.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~281s 2024-03-04T19:12:08.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~281s 2024-03-04T19:12:08.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~281s 2024-03-04T19:12:08.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~281s 2024-03-04T19:12:08.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~281s 2024-03-04T19:12:08.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~281s 2024-03-04T19:12:08.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~281s 2024-03-04T19:12:08.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:08.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~280s 2024-03-04T19:12:08.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~280s 2024-03-04T19:12:08.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~280s 2024-03-04T19:12:08.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~280s 2024-03-04T19:12:08.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2024-03-04T19:12:08.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2024-03-04T19:12:08.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~280s 2024-03-04T19:12:11.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:11.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:11.452 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:11.452 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:12:11.452 INFO:tasks.ceph:Archiving crash dumps... 2024-03-04T19:12:11.455 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264/remote/smithi027/crash 2024-03-04T19:12:11.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:11.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:13.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~286s 2024-03-04T19:12:13.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~286s 2024-03-04T19:12:13.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~285s 2024-03-04T19:12:13.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~285s 2024-03-04T19:12:14.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:18.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~291s 2024-03-04T19:12:18.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~291s 2024-03-04T19:12:18.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~291s 2024-03-04T19:12:18.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~291s 2024-03-04T19:12:18.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~291s 2024-03-04T19:12:18.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~291s 2024-03-04T19:12:18.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~291s 2024-03-04T19:12:18.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~290s 2024-03-04T19:12:18.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~290s 2024-03-04T19:12:18.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~290s 2024-03-04T19:12:18.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~290s 2024-03-04T19:12:18.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~290s 2024-03-04T19:12:18.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~290s 2024-03-04T19:12:18.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~290s 2024-03-04T19:12:23.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~296s 2024-03-04T19:12:23.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~296s 2024-03-04T19:12:23.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~296s 2024-03-04T19:12:23.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~296s 2024-03-04T19:12:23.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~296s 2024-03-04T19:12:23.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~296s 2024-03-04T19:12:23.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~296s 2024-03-04T19:12:23.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:23.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~295s 2024-03-04T19:12:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~295s 2024-03-04T19:12:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~295s 2024-03-04T19:12:23.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~295s 2024-03-04T19:12:23.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~295s 2024-03-04T19:12:23.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~295s 2024-03-04T19:12:23.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~295s 2024-03-04T19:12:26.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:28.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~301s 2024-03-04T19:12:28.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~301s 2024-03-04T19:12:28.246 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~300s 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~300s 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~300s 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~300s 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~300s 2024-03-04T19:12:28.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~300s 2024-03-04T19:12:28.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~300s 2024-03-04T19:12:28.919 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-04T19:12:38.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:38.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:41.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:44.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:44.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:44.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:44.732 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:12:44.732 INFO:tasks.ceph:Compressing logs... 2024-03-04T19:12:44.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:44.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:47.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:12:56.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:12:56.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:12:59.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:11.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:11.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:14.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:17.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:17.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:18.016 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:18.016 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:13:18.017 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/ceph.py", line 264, in ceph_log ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:13:18.042 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T19:13:18.060 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:13:18.076 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T19:13:18.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:18.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:21.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:30.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:30.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:33.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:45.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:45.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:48.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:51.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:51.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:51.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:13:51.356 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:13:51.408 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:13:51.425 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T19:13:51.435 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T19:13:51.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:13:51.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:13:54.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:03.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:03.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:06.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:18.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:18.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:21.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:24.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:24.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:24.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:24.700 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:14:24.700 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:14:24.712 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T19:14:24.722 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T19:14:24.723 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T19:14:24.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:24.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:27.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:36.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:36.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:39.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:51.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:51.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:54.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:57.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:57.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:14:58.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:14:58.012 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:14:58.012 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:14:58.020 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T19:14:58.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T19:14:58.045 INFO:teuthology.task.internal:Duration was 11592.628777 seconds 2024-03-04T19:14:58.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T19:14:58.115 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T19:14:58.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:14:58.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:01.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:10.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:10.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:13.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:25.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:25.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:28.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:31.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:31.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:31.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:31.388 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:15:31.388 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:15:31.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T19:15:31.412 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T19:15:31.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:31.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:34.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:43.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:43.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:15:46.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:15:58.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:15:58.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:01.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:04.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:04.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:04.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:04.700 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:16:04.701 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:16:04.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T19:16:04.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:04.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:07.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:16.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:16.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:19.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:31.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:31.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:34.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:37.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:37.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:38.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:38.012 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:16:38.012 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:16:38.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T19:16:38.037 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T19:16:38.039 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264/remote/smithi027 2024-03-04T19:16:38.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:38.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:41.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:16:50.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:16:50.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:16:53.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:05.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:05.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:08.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:11.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:11.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:11.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:11.325 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:17:11.325 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:17:11.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T19:17:11.351 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T19:17:11.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T19:17:11.366 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T19:17:11.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:11.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:14.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:23.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:23.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:26.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:38.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:38.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:41.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:44.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-04T19:17:44.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T19:17:44.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-04T19:17:44.636 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-04T19:17:44.637 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 500, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 327, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a5335f5f0d4a33387826931c0d9be1a7ec43d970/qa/tasks/rgw.py", line 258, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi027 2024-03-04T19:17:44.652 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T19:17:44.664 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T19:17:44.677 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionLostError: SSH connection to smithi027 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-03-04T19:17:44.678 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T19:17:44.709 DEBUG:teuthology.nuke:shortname: smithi027 2024-03-04T19:17:44.709 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T19:17:44.734 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU='} 2024-03-04T19:17:44.735 ERROR:teuthology.nuke:Could not nuke {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU='} Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 294, in nuke_one nuke_helper(ctx, should_unlock, keep_logs, should_reboot) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 318, in nuke_helper check_lock.check_lock(ctx, None, check_up=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi027.front.sepia.ceph.com is not locked 2024-03-04T19:17:44.746 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETBVcNTW9+Ib9bqGQ0aMi/GcAWhzRxdo6CoEexjs/9Ayez/Y6mPpD68hJLt9RAxz+poGdra1VSDTYJOHyXZdPU= 2024-03-04T19:17:44.751 DEBUG:teuthology.nuke:shortname: smithi071 2024-03-04T19:17:44.751 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T19:17:44.755 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} duration: 11592.628776788712 failure_reason: rgw multisite test failures flavor: default owner: scheduled_suriarte@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=14828d6955c7492784b96dc5a3afd0e5 status: fail success: false 2024-03-04T19:17:44.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T19:17:44.780 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/suriarte-2024-03-04_15:42:14-rgw-main-distro-default-smithi/7581264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI='} 2024-03-04T19:17:44.780 ERROR:teuthology.nuke:Could not nuke {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEpNobZ1bclrsrvguMJic9yc7rYH4H2X89jCk2fG76a7SPYnRZtyDKSQBpu4pU4qcjzFgvSrcLuNnfxIo2nHBmI='} Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 294, in nuke_one nuke_helper(ctx, should_unlock, keep_logs, should_reboot) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/nuke/__init__.py", line 318, in nuke_helper check_lock.check_lock(ctx, None, check_up=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi071.front.sepia.ceph.com is not locked 2024-03-04T19:17:44.829 INFO:teuthology.run:FAIL